Author Topic: International boards, stick to the basics firmware-wise  (Read 2838 times)

0 Members and 1 Guest are viewing this topic.

Offline Nlight

  • Thread Starter
  • Posts: 57
  • Location: France
International boards, stick to the basics firmware-wise
« on: Wed, 26 September 2018, 09:10:39 »
Hi, if you are new to making your own board, and if you are in a country with a different layout than the regular qwerty, you may be tempted to to make some layers to handle punctuations and such, right ? You may be tempted to arrange your firmware mapping to fit the french, german, whatever layout you are used to, right ?

Well, IMO this is a BAD idea1  >:D

What are the problem you will face :
  • while designing you firmware, you will have to have two layout in front of you : a qwerty layout, and the one you want to achieve (may it be customized)
  • you will have to pin point the aimed character on the qwerty, to translate it as eventually a combination of keys into your layout
  • to actually yield the same characters when you press shift, especially punctuation, you will have to replace the shift modifier with a function that turns on a layer, that will hold your "shifted characters and punctuations", but that means that any app that uses the Shift key will pose a problem.
  • for some character that you won't find on a qwerty, you will have to know its unicode pattern, and translate it as a macro in your layout, keep in mind that these macro works only on one OS, as each three major OS have their own way to do it, which means that if you want your board to work anywhere, you will have to have three toggle keys mutually exclusive, one for each OS
  • you will need the machine you plug your keyboard into, to be set to US english, which might work in an english speaking country, but in your country, chances are it won't be that.

Lots of hassle, lots of  conditions, and pretty weird behavior to an uninitiate to your board.

In the firmware, it is highly advisable to stick to a qwerty mapping, eventually take advantage of layers to play with keys that you would not be able to remap otherwise, like for example arrow keys replacing some letters in a layer. You can also take advantage of other advanced functions, BUT NO ALPHA KEYS (or puctutation) out of its regular place.

Aim for this in your firmware :


Aim for this kind of layout in your OS :



"Why making a qwerty board ? I am not american, I don't understand"

Well, it does not matter, a keyboard does not send characters to your computer, it sends scancodes, and each scancodes is supposed to represent a precise key physical location in a standard US keyboard2. These scan codes are then translated by your OS into characters according to the language and layout you chose. For example in azerty boards, A and Q are swapped comparing to qwerty, in azerty "A" key emits the scancode 0x10, in qwerty "Q" key emits 0x10.
You see ? keyboards are language independent, just keep that in mind.

Hope it helps new builders, cheers  :)

[1] : there may be pros, and some specific situations where it wight be desirable, but I yet have to come upon one, if you do, please say so.
[2] : In fact some boards around the world implements more keys than a standard US, but these additional keys have their own scancodes, and the common keys still emits the very same scancodes.

EDIT (2018-10-06) : after trying to deal with unicode in QMK for a whole day, I can tell with confidence : this is a mess not uniformly handled throughout all three major os, and it does require you to tinker in the said OS, therefore I DEFINITELY stand by my point that it is a WHOLE LOT easier and quicker to make a qwerty firmware, and deal with accents, punctuation and such in a layout inside the OS ! Even in linux
« Last Edit: Sat, 06 October 2018, 06:45:58 by Nlight »

Offline Findecanor

  • Posts: 5035
  • Location: Koriko
Re: International boards, stick to the basics firmware-wise
« Reply #1 on: Wed, 26 September 2018, 11:38:47 »

Offline Nlight

  • Thread Starter
  • Posts: 57
  • Location: France
Re: International boards, stick to the basics firmware-wise
« Reply #2 on: Wed, 26 September 2018, 12:55:21 »
 :))

Offline algernon

  • Posts: 311
  • A tiny mouse, a hacker.
    • Diaries of a Madman
Re: International boards, stick to the basics firmware-wise
« Reply #3 on: Wed, 26 September 2018, 17:53:09 »
to actually yield the same characters when you press shift, especially punctuation, you will have to replace the shift modifier with a function that turns on a layer, that will hold your "shifted characters and punctuations", but that means that any app that uses the Shift key will pose a problem.

Using a shift layer is only one of the ways you can achieve this, but it isn't the only way. You can have macros that check the shift state, and input the appropriate sequence accordingly. This way you won't lose the ability to hold shift, and have it be sent to the host. This is actually a much more reliable way - this is how I write Hungarian at ~70WPM. And no, the size of the macros will not become unwieldy. I used to use a similar method to turn Shift+1 into $ for example, that also worked perfectly fine. Granted, this could be sometimes surprising, like when games interpret Shift+4 not as a way to input $, but as two distinct keys chorded together. I kept a QWERTY layer around for these cases - which made sense because I didn't want to remap the key bindings of every single game either. This, by the way, is a legit problem if you set your OS layout to something other than QWERTY, and is not unique to customised firmware.

for some character that you won't find on a qwerty, you will have to know its unicode pattern, and translate it as a macro in your layout, keep in mind that these macro works only on one OS, as each three major OS have their own way to do it, which means that if you want your board to work anywhere, you will have to have three toggle keys mutually exclusive, one for each OS

Or you can use USB fingerprinting to detect the OS (works for the three major ones, though it can't tell iOS and OSX apart). Or you can use a LEADER sequence to select which OS you want to support. Or banish the toggle keys to a layer (why is three toggle keys a problem by the way?). Or - and this is what I'd recommend - you can configure your OSes to use the same unicode input method. For example, you can use WinCompose under Windows, and then you can use the same sequence as under Linux - no need to toggle!

you will need the machine you plug your keyboard into, to be set to US english, which might work in an english speaking country, but in your country, chances are it won't be that.

Changing the layout to US English is a lot easier than tweaking your keyboard firmware, so anyone who previously considered improving their firmware to be able to input international symbols, or even emoji, are most likely able to switch to US English on their OS.

Granted, if you plug your keyboard into random computers, then sticking to basics is a good idea. If you only plug it into computers you control, then do go wild. It's a liberating feeling.

For the record, on my own layout, pretty much no key is left in place. Yet, it works just fine under Linux and Windows, including Hungarian symbols and even Unicode emojis, and they do not hinder my typing speed, nor the reliability of the keyboard.

In short, go ahead and be crazy. Just pay attention to the gotchas, and decide yourself if they're worth it. For me, they did.

Offline Nlight

  • Thread Starter
  • Posts: 57
  • Location: France
Re: International boards, stick to the basics firmware-wise
« Reply #4 on: Thu, 27 September 2018, 04:57:08 »
well... Thank you mate, your position allows to show the pros and cons of two different philosophies :)

This is informative, i've learned some things, but I still stand by my point. However your last point, and probably the most important holds completely true:
Quote
If you only plug it into computers you control, then do go wild. It's a liberating feeling.
Even though "liberative" is relative to all the hassle you had to go through... :rolleyes:  ;) :))

Quote
Using a shift layer is only one of the ways you can achieve this, but it isn't the only way. You can have macros that check the shift state, and input the appropriate sequence accordingly. This way you won't lose the ability to hold shift, and have it be sent to the host. This is actually a much more reliable way - this is how I write Hungarian at ~70WPM. And no, the size of the macros will not become unwieldy. I used to use a similar method to turn Shift+1 into $ for example, that also worked perfectly fine. Granted, this could be sometimes surprising, like when games interpret Shift+4 not as a way to input $, but as two distinct keys chorded together. I kept a QWERTY layer around for these cases - which made sense because I didn't want to remap the key bindings of every single game either. This, by the way, is a legit problem if you set your OS layout to something other than QWERTY, and is not unique to customised firmware.
You are making my point exactly : unexpected behavior, lot of work for a workaround, and a less readable firmware code. If you intend for someone other than yourself to use your keyboard, the simpler it is the better.

Quote
Or you can use USB fingerprinting to detect the OS (works for the three major ones, though it can't tell iOS and OSX apart).
This , I had no idea and this is interesting, it does not change my mind, but it is worth learning.

Quote
Or banish the toggle keys to a layer (why is three toggle keys a problem by the way?)
Maybe, but the more layers, the more complex, the bigger the learning curve, and also the more layer switches on your main layer. You could argue that you don't need to make one layer specifically for that and integrate it to another... Yes, but the more different functional theme inside a layer, the harder to remember.

Quote
Changing the layout to US English is a lot easier than tweaking your keyboard firmware
Yes, exactly ! It is also a lot easier to make your own mapping inside the OS than tweaking your firmware.

Quote
so anyone who previously considered improving their firmware to be able to input international symbols, or even emoji, are most likely able to switch to US English on their OS
True, but it still is faster and easier to make your mapping inside your OS thanks to great GUI tools, and ship it on a USB stick, or repository on the net (or better, a partition inside your keyboard, but this is not the simplest thing to do)

Quote
Granted, if you plug your keyboard into random computers, then sticking to basics is a good idea.
Yes, my whole argumentation revolves mainly around the fact that you can give/sell your keyboard, or that your are a new builder and would more likely be quickly successful with simple approach.

« Last Edit: Thu, 27 September 2018, 05:29:42 by Nlight »

Offline vvp

  • Posts: 886
Re: International boards, stick to the basics firmware-wise
« Reply #5 on: Thu, 27 September 2018, 12:23:33 »
People put advanced functionality(*) into keyboards so that it works in each OS without need to install anything.

You argue that keyboard should be kept stupid qwerty. That is a reasonable approach. Put all the special functionality into your special keyboard driver. Implement it for Windows, Linux and Mac at least and you are golden. Your driver will make any stupid off-the-shelf keyboard feature rich.


(*) Remapping, macro support, configurations, ... You are thinking too much about regional language support. That is only a small part of it. But there is some portability even for Unicode input between Windows and Linux. See Compose and Hexadecimal code input. You still may need a configuration option to initiate the sequence differently for each OS.

Offline algernon

  • Posts: 311
  • A tiny mouse, a hacker.
    • Diaries of a Madman
Re: International boards, stick to the basics firmware-wise
« Reply #6 on: Thu, 27 September 2018, 14:17:10 »
Quote
If you only plug it into computers you control, then do go wild. It's a liberating feeling.
Even though "liberative" is relative to all the hassle you had to go through... :rolleyes:  ;) :))

There were no hassles, really. There was one compromise I made, that I'll have a QWERTY layer for games. But that's about as much a hassle as having a QWERTY layout on the OS side to switch to - for the same reasons.

You are making my point exactly : unexpected behavior, lot of work for a workaround, and a less readable firmware code. If you intend for someone other than yourself to use your keyboard, the simpler it is the better.

It is only unexpected until you figure out how keyboards work. Then it makes perfect sense. The firmware remains readable too:

Code: [Select]
if (isShiftPressed()) {
  // return one macro
} else {
  // return other macro
}

Depending on what you want to do (international symbols or changing what symbol is input for a key pressed with shift), this can be further abstracted away. For example, you might end up with `KC_HUN_AA` in your keymap, with no other code in your firmware. The core firmware (or a plugin, extension, library, whatever) could provide the small amount of code that turns this into the key sequence that's needed to input á. Thus, the keymap becomes as simple as a QWERTY keymap. Yes, the firmware does a bit more under the hood, but it already does a whole lot of things you have no idea about.

Quote
Changing the layout to US English is a lot easier than tweaking your keyboard firmware
Yes, exactly ! It is also a lot easier to make your own mapping inside the OS than tweaking your firmware.

Err, no. If I change my mapping in the firmware, it will work under both Linux, Windows, and whatever I plug it into (at least most of it). If I rely on the host, I have to change the layout on every single machine, for every OS I use. That's... not a prospect I'm looking for.

Quote
so anyone who previously considered improving their firmware to be able to input international symbols, or even emoji, are most likely able to switch to US English on their OS
True, but it still is faster and easier to make your mapping inside your OS thanks to great GUI tools, and ship it on a USB stick, or repository on the net (or better, a partition inside your keyboard, but this is not the simplest thing to do)

You can also change the layout by using a GUI, and just flash the resulting hex file. Basically, instead of carrying an USB stick with the layout, for N+1 OSes, you carry a single hex file, and a flasher app for N+1 OSes. It's pretty much the same effort. With the advantage that you don't have to install the mapping everywhere. You only do that when you reconfigure your layout, and once you are comfortable with one, that happens rarely.

Quote
Granted, if you plug your keyboard into random computers, then sticking to basics is a good idea.
Yes, my whole argumentation revolves mainly around the fact that you can give/sell your keyboard, or that your are a new builder and would more likely be quickly successful with simple approach.

If you want to sell your board, it's a good idea to flash the default firmware anyway. Using a custom one does not, in any way, make it harder to sell it. If you loan it, or want someone else to use it, have a QWERTY layout on it, and that other person can still use the OS-side mapping. This way, you can both use advanced features of the firmware (oh, btw, good luck implementing something like OneShots or TapDance on the host side, reliably, across all OSes and programs!), and still be able to use the same board at random computers, or give the keyboard to someone else after switching to your QWERTY layer. You can even make it so that you store the default layer in EEPROM, so even if they unplug & replug the board, it will still remain in QWERTY.

Offline Nlight

  • Thread Starter
  • Posts: 57
  • Location: France
Re: International boards, stick to the basics firmware-wise
« Reply #7 on: Thu, 27 September 2018, 16:57:22 »
This is great stuff and debunk you're making here, I love it  :D

Even if I still think it ends up being too much work, and not the easiest either, I must admit you make a lot of sense.

I want to argue things, but I can't, I just can't add anything new to this. Though I remain very skeptical when you say it does not impact sales... I mean, if make a hungarian board, well your primary target is hungarians or ths who study this language (I guess), while a basic firmware with not many fancy things in the frimware can target a whole lot more people who exactly knwo what to expect, but this is just a guess. Now I suppose it remains a question of preference, also that there are a lot more docs for making os mappings, than firmware mapping, and that it remains easier to do (thanks to guis), no compiling, no flashing (while this is easier than one would think, it is daunting to most people).

But hat down for a well made case mate.

There is one thing I am really curious about, it's the thing about how your keyboard actually comes to know the os it is connected to... would you mind to explain it or point me to some resources ?

Offline Nlight

  • Thread Starter
  • Posts: 57
  • Location: France
Re: International boards, stick to the basics firmware-wise
« Reply #8 on: Thu, 27 September 2018, 17:15:09 »
Vvp, sorry I did not notice your post... developping drivers ? the hell if I know how to do that... can you too point me in the right direction ? I love learning stuff.
(edit : ok, plenty of sources, no problem, i mean, no problem finding resources)

And what is this Compose stuff ?
(edit : ok, unicode input method in Linux)
« Last Edit: Fri, 28 September 2018, 05:16:02 by Nlight »

Offline algernon

  • Posts: 311
  • A tiny mouse, a hacker.
    • Diaries of a Madman
Re: International boards, stick to the basics firmware-wise
« Reply #9 on: Wed, 10 October 2018, 14:53:54 »
There is one thing I am really curious about, it's the thing about how your keyboard actually comes to know the os it is connected to... would you mind to explain it or point me to some resources ?

There are a number of ways. Some OSes are reasonably simple to fingerprint from the firmware side, detecting Linux for example is reasonably easy. OSX isn't particularly hard either. For a few hints, look at FingerprintUSBHost. Another, much more reliable option is to have a small tool on the host tell the keyboard what OS is running. The keyboard can set up a virtual serial port for this purpose, and use a simple protocol to allow the host to talk to the keyboard (and vice versa). This is what Kaleidoscope does via the FocusSerial plugin.