The context here is a SINGLE LAYER within Easy Keymap. For purposes of this description I will use a 60% keyboard like the GH60. The GH60 has five rows (0-4) and fifteen columns (0-14). The keys in the GH60 matrix can be described using layout coordinates of (row,column) so the upper left key (Escape or Tilde) is (0,0) and the Space bar is something like (4,5). For the purposes of pseudo-code, I will create an imaginary function called KEYMAP() that takes three arguments, row, column, and scancode assignment.
So this is how to make that upper-left key be escape:
KEYMAP(0,0) == HID_KEYBOARD_SC_ESCAPE
The latest versions of Easy Keymap extend this by adding modifiers to the assignment as shown here:
KEYMAP(0,0) == HID_KEYBOARD_SC_LEFT_SHIFT + HID_KEYBOARD_SC_GRAVE_ACCENT_AND_TILDE
I am proposing the modifiers be moved to the other side of the assignment operator as shown here. The first line shows what an unmodified keypress delivers. The second line, which I will call "IFMOD" shows what is assigned to KEYMAP(3,0) IFMOD (RSHIFT) is active.
KEYMAP(3,0) == HID_KEYBOARD_SC_LEFT_SHIFT # Unchanged from current
KEYMAP(3,0,RSHIFT) == HID_KEYBOARD_SC_NUM_LOCK # Additional argument added
Here's how it currently works (simplified):
The mapping function takes three arguments, and returns four things
KEYMAP(row, col, layer) -> (scancode, modifiers, mode, tapkey)
Supplying those four results takes 3 bytes per matrix location, and is maxing out memory usage
So consider I mapped Fn+Del to be the same as CTRL+ALT+DEL. It would look like this:
KEYMAP(2, 14, 0) -> (HID_KEYBOARD_SC_DELETE, 0, 'normal', 0)
KEYMAP(2, 14, 1) -> (HID_KEYBOARD_SC_DELETE, (Ctrl|Alt), 'normal', 0)
Or if I mapped Fn+Esc to be my flip key, it would look like this:
KEYMAP(0, 0, 0) -> (HID_KEYBOARD_SC_ESCAPE, 0, 'normal', 0)
KEYMAP(0, 0, 1) -> (SCANCODE_BOOT, 0, 'normal', 0)
Or if I mapped a SpaceFn layout, I might have something like this:
KEYMAP(5, 6, 0) -> (SCANCODE_FN, 0, 'tapkey', HID_KEYBOARD_SC_SPACE)
Where am I going with this?
When you look at it in this context, I think what you're actually proposing is a shift layer. What I'm saying is if you just rename "shift layer" to "Layer 3", you've already got it.
Actually I haven't even finished reading your post yet, so I could be missing important stuff. I'm busy lately, but I wanted to respond.