Designing a new layout. Not very scientifically.

[Zustiur]

I've been using Colemak on my ErgoDox for about 9 months. Prior to that I'd never left QWERTY. My reading this weekend has taken me to a number of topic all of which lead me to be curios. I finally looked up the alternative layouts (including odd ones like BU-Teck) and read the reasonably convincing logic behind the workman layout.

However, all of these layouts seem to assume the standard keyboard. Workman at least provides a weighting diagram for grid/matrix boards but assumes all peripheral keys will remain the same. When I noticed that this continued to give more preference to comma than to full stop I immediately felt that was a mistake. I then wondered about frequency of punctuation in English and felt someone should at least attempt to intermingle the keys according to frequency. Even if it's purely an intellectual excercise/thought experiment that never makes it into an actual board.

That lead me to these wiki articles:
https://en.wikipedia.org/wiki/Letter_frequency
https://en.wikipedia.org/wiki/Punctuation_of_English

I combined their findings with a few assumptions to produce this combined list:

Rank	Key	Keystrokes/1000
0	Space	130
1	E	127.02
2	T	90.56
3	A	81.67
4	O	75.07
5	I	69.66
6	N	67.49
7	.	65.3
8	S	63.27
9	,	61.6
10	H	60.94
11	R	59.87
12	'"	51
13	D	42.53
14	L	40.25
15	Shift	33
16	Shift	33
17	C	27.82
18	U	27.58
19	M	24.06
20	W	23.61
21	F	22.28
22	G	20.15
23	Y	19.74
24	P	19.29
25	-	15.3
26	B	14.92
27	V	9.78
28	K	7.72
29	;:	6.6
30	!	3.3
31	J	1.53
32	X	1.5
33	Q	0.95
34	Z	0.74

My assumptions were:

• Space is used 130 times per 1000 key strokes based on a statement in the wiki article 'slightly more frequent than e'. From other web pages I think 150 would have been more accurate, but that doesn't matter really. Either way it's the most frequent key.
• Space should be on the thumbs and treated as one finger, even though the workload can be halved between the thumbs.
• Upper case letters and symbols require the use of shift a bit more frequently than full stop (period).
• Because shift would be required on both hands to avoid clashing with other keys on the same finger, I have halved its freqency and entered it twice
• I could not find a comparison of number freqency to letter frequency, so I've left those out of the equasion. In fact I couldn't find a general frequency of numbers, only specific cases like telephone numbers or in pi. I therefore ignored them on the assumption that there'd still be a dedicated row of numbers in numeric order from left to right.
• Finally, I gave preference to the right hand over the left because I use a left handed mouse. Right handers should flip the whole layout

Determining the layout, I started with workman's weighting for grid boards:

4	2	2	3	4	4	3	2	2	4
1.5	1	1	1	3	3	1	1	1	1.5
4	4	3	2	4	4	2	3	4	4

Then broke it down into much finer detail based purely on my own preference:

	26	14	12	20	30	29	19	11	13	25
32	8	6	4	2	18	17	1	3	5	7	31
34	24	22	16	10	28	27	9	15	21	23	33

Then let excel place the keys for me:

	B	L	'"	W	!	;:	M	R	D	-
X	S	N	O	T	U	C	E	A	I	.	J
Z	P	G	Shift	H	K	V	,	Shift	F	Y	Q

For pure typing, that's probably quite good. However, we are using computers, not typing on electronic typewriters. Therefore additional elements should be taken into account. For my second version there are some additional assumptions:

• We should attempt to account for the increased frequency of v, c, x, z, b, I and y due to paste, copy, cut, undo, bold, itatlic and redo respectively.
• I assigned an increase of 1-10 per 1000 for each of these keys. Unfortunately I have no data on the frequency of these so this is gut feel
• I also increased shift by one rank so that it would end up on the same finger position for both hands, but also because it is used in some command chords like shift-tab

In most cases this just shifts the priority of a given key by one place.

The resulting matrix is:

	P	L	'"	W	;:	K	M	R	D	V
!	S	N	O	T	U	C	E	A	I	.	X
Q	B	F	Shift	H	Z	-	,	Shift	Y	G	J

This still doesn't account for frequency of Enter, Backspace, Ctrl, Alt, Win, App, Tab, Backspace, ? and various other keys. Nor does it do anything about avoiding common same finger bigrams. However, it looks interesting enough to make me chuck it into KLE. A board built with this layout might look something like this:

More

["!\n1","@\n2","#\n3","$\n4","%\n5","^\n6",{w:2},"Escape","&\n7","*\n8","(\n9",")\n0","_\n-","+\n="],
["?\n/","P","L","\"\n'","W",":\n;",{w:2},"App","K","M","R","D","V","|\n\\"],
["J",{c:"#65af4c"},"S","N","O","T",{c:"#cccccc"},"U",{w:2},"Tab","C",{c:"#65af4c"},"E","A","I",">\n.",{c:"#cccccc"},"X"],
["O","B","F","Shift","H","Z",{w:2},"Win","_\n-","<\n,","Shift","Y","G","Q"],
["Ctrl","Alt","Enter","Del",{w:2},"Space",{w:2},"BackSpace",{w:2},"Space","Del","Enter","Alt","Ctrl"]

I can't help but call this the SNOT layout thanks to that word being the home keys for the left hand.


I'm extremely interested to get other peoples views on this layout, on the steps I took to generate it etc.
I would also love to be pointed to an analyzer that will calculate the bigrams and same-finger-units for me. The workman designer must have used one but I cannot find a link.

[Pro XKB]

finally looked up the alternative layouts (including odd ones like BU-Teck) and read the reasonably convincing logic behind the workman layout.

However, all of these layouts seem to assume the standard keyboard.

BU-Teck was explicitly made for the "Truly ergonomic keyboard", not a standard keyboard.

I would also love to be pointed to an analyzer that will calculate the bigrams and same-finger-units for me.

You could use the AdNW tool from here:
 
  https://sites.google.com/site/ausderneowelt

It also comes with frequency files, including punctuation, space, upper and lower case letters separate.

[hoggy]

There's a scale invariant law about the frequency of numbers that after some assumptions, one is more frequent than two, three and so on. Can't think of the name of the law, but it could be used to detect made up numbers in financial accounts.

From memory (as good as it's proving to be), didn't programmer dvorak attempt to optimise number placements?

[stevep]

There's a scale invariant law about the frequency of numbers that after some assumptions, one is more frequent than two, three and so on. Can't think of the name of the law..

Sounds like you mean Benford's Law.

[Zustiur]

BU-Teck was explicitly made for the "Truly ergonomic keyboard", not a standard keyboard.

That is correct. I didn't mean to imply otherwise. Oops.

You could use the AdNW tool from here:
 
  https://sites.google.com/site/ausderneowelt

It also comes with frequency files, including punctuation, space, upper and lower case letters separate.

Thanks. I had a look at the documentation but I'm too frazzled to make sense of it today. I'll try again later.

[jacobolus]

This is a reasonable start, but do note that what you care about ultimately is the ease of typing groups of 3–4 letters, not only single letters.

Also, you may need to sometimes type multiple-modifier shortcuts, some of which I suspect will end up quite awkward. (Though they’re also very awkward on standard keyboards.)

Are you planning to make a keyboard in a grid like that, or is that just an abstract representation? Some of the keys there aren’t going to be all that reachable, and the thumbs are pretty under-used.

[lydell]

Hi, Zustiur. I’ve created my own layout, anishtro, so I’ve got some experience to share.

I've been using Colemak on my ErgoDox for about 9 months. [...]

However, all of these layouts seem to assume the standard keyboard.

Yes, most layouts are made for standard keyboards. Creating a layout is not a one-size-fits all thing in two ways. A layout does not fit every person, because we all have different opinions and think different things are comfortable. A layout does not either fit every keyboard, because when creating a layout you must keep the physical constraints of your keyboard in mind.

Workman at least provides a weighting diagram for grid/matrix boards but assumes all peripheral keys will remain the same. When I noticed that this continued to give more preference to comma than to full stop I immediately felt that was a mistake.

That means that you and the author of Workman disagree. If so, Workman is not a good choice for you.

Even if it's purely an intellectual excercise/thought experiment that never makes it into an actual board.

If you think it is fun then by all means go for it! Even if you will be the only one using the end result it doesn’t matter as long as you like it and had fun along the way. This is the case for me: I think I might be the only one in the world using the anishtro layout (but then you should keep in mind that I didn’t even share it on geekhack until yesterday, even though I’ve used the layout for one and a half years).

I then wondered about frequency of punctuation in English and felt someone should at least attempt to intermingle the keys according to frequency.

That lead me to these wiki articles:
https://en.wikipedia.org/wiki/Letter_frequency
https://en.wikipedia.org/wiki/Punctuation_of_English

I also used those Wikipedia articles in the beginning, but I find them lacking. They’re not enough to do research on. Instead, I recommend getting a large text corpus that you think is representative and calculate frequencies from that yourself. That’s what I did: https://github.com/lydell/anishtro/blob/master/how.md#getting-data (also see the following section).

I combined their findings with a few assumptions to produce this combined list:

I also tried to think logically and make assumptions in the beginning. After I had gotten real data I realized how bad I was making assumptions. Mostly they were wrong.

I gave preference to the right hand over the left because I use a left handed mouse. Right handers should flip the whole layout
[...]
Determining the layout, I started with workman's weighting for grid boards:
[...]
Then broke it down into much finer detail based purely on my own preference:

That’s the way to go! Nail down the things you like and optimize for them!

Then let excel place the keys for me:

I’d be very interested in how you managed to get a spreadsheet program to place the keys for you.

• We should attempt to account for the increased frequency of v, c, x, z, b, I and y due to paste, copy, cut, undo, bold, itatlic and redo respectively.
• I assigned an increase of 1-10 per 1000 for each of these keys. Unfortunately I have no data on the frequency of these so this is gut feel

[...]
This still doesn't account for frequency of Enter, Backspace, Ctrl, Alt, Win, App, Tab, Backspace, ? and various other keys.

My gut feel often proved to be wrong when it came to saying how common certain letters are. If you really want to take all those things in consideration I’d recommend running a key logger for a month or so and then analyze the results.

Nor does it do anything about avoiding common same finger bigrams.

For me, that’s one of the most important characteristics of a keyboard layout, because that’s what bothered me the most in QWERTY. If I were you, I’d look into minimizing it. You have a few pairs I’d dislike, such as ME, TH, AR and CK. On the other hand you’ve also put some uncommon combinations on the same finger, which is good, such as LN and IY. You’ve also “protected” many vowels with punctuation characters, other vowels and uncommon letters (such as Z), which is wise since vowels are most likely to be part of common letter pairs.

I can't help but call this the SNOT layout thanks to that word being the home keys for the left hand.

Naming layouts after sequences of letters in them, especially on the home row, is common.

I would also love to be pointed to an analyzer that will calculate the bigrams and same-finger-units for me.

In my opinion it does not make much sense to use somebody else’s analyzer, because you will most likely have different opinions and find that the tool analyses things you aren’t interested in. It is better to create your own tools (but you could of course base it on your own). I’ve included the tools I used for anishtro. Just for the fun of it, here’s how your layout scores in my analyzer (which uses my opinions, not yours):

Code: [Select]

{ sameFinger:
   [ [ 'ht', 3.1143912946997787 ],
     [ 'ar', 2.099149674202991 ],
     [ 'em', 1.1139997279205736 ],
     [ 'ce', 1.043975483884984 ],
     [ 'di', 0.7793704826926966 ],
     [ 'tu', 0.5794360714213668 ],
     [ 'hw', 0.25737627498436577 ],
     [ 'ek', 0.2437593721275835 ],
     [ 'ps', 0.23336242065858737 ],
     [ 'ck', 0.169687558676825 ],
     [ 'fl', 0.10791363185170741 ],
     [ 'tw', 0.09908139571573184 ],
     [ 'hu', 0.09840895606848332 ],
     [ 'bs', 0.07113635575988525 ],
     [ 'ln', 0.06022214302377488 ],
     [ 'dy', 0.05717030154780089 ],
     [ 'fn', 0.05559265468310247 ],
     [ 'iy', 0.032070198561082605 ],
     [ 'cm', 0.019332639858394556 ],
     [ 'km', 0.01761274614523972 ],
     [ 'tz', 0.01516868665812496 ],
     [ 'uz', 0.014560904669265733 ],
     [ 'bj', 0.010202978493828297 ],
     [ 'hz', 0.006595081155706504 ],
     [ 'bp', 0.004254473922014588 ],
     [ 'uw', 0.0035949658064439375 ],
     [ 'js', 0.0020302504734233747 ],
     [ 'gv', 0.0013448792944970125 ],
     [ 'jp', 0.0013319477628191568 ],
     [ 'vx', 0.000827618027382777 ],
     [ 'wz', 0.0008017549640270651 ],
     [ 'gx', 0.00034915135530210904 ],
     [ 'gq', 0.00032328829194639726 ],
     [ 'qx', 0.0003103567602685414 ],
     [ 'qv', 0.00023276757020140602 ] ],
  rolls:
   [ [ 'no', 2.1511473630796494 ],
     [ 'ot', 1.3226111969477448 ],
     [ 'ai', 0.7833404629177985 ],
     [ 'ae', 0.7195621486826131 ],
     [ 'ns', 0.49520007407181343 ],
     [ 'dr', 0.34318991919861747 ],
     [ 'mr', 0.1994430130675714 ] ],
  sameFingerTotal: 10.314978489690207,
  rollsTotal: 6.014494177965807 }

(You can compare with QWERTY which gets 7.78% same finger usage and 6.67% rolls, or anishtro which gets 0.89% same finger usage and 14.65% rolls. So for me, your layout would suck But remember that we all have different opinions, which might make it rock for you.)

It looks like that layout image you posted does not really correspond to the text versions you wrote, and it looks like some letters are duplicated in the image.

[Zustiur]

Are you planning to make a keyboard in a grid like that, or is that just an abstract representation? Some of the keys there aren’t going to be all that reachable, and the thumbs are pretty under-used.

That is just an abstract. See my making stuff together thread for my actual keyboard shape designs. This thread is more about theory and curiosity. It's a speculation on what could have been the standard if qwerty wasn't so dominant at the beginning of computing.

[Zustiur]

Thanks for the input lydell. I feel that the only 'true' way to design a keyboard layout is to spend a lot of time with keyloggers on your machines. To create a layout for the masses would require logging the activity and usage of many people.
Sadly installing a logger at my workplace is out of the question.

The trouble with using a corpus is that it won't respect everyday computer usage.

[Zustiur]

Then let excel place the keys for me:

I’d be very interested in how you managed to get a spreadsheet program to place the keys for you.

It's pretty easy if you're familiar with the vlookup function.
I essentially had 3 tables:
The first table was 3 columns: rank, key, keystrokes/1000
The second table was laid out like a keyboard with each key position given a unique number to represent the key's rank. i.e. key 1 is right hand index finger home position.
The third table is shaped like the second but contains vlookup formulae like this:
=VLOOKUP(M49,$A$43:$B$77,2,FALSE)
Where M49 is the corresponding position in the table 2 and A43:B77 is table 1.
So in English the formula says "the value of THIS cell equals position number M49 (which is key rank 1) in the letter ranking table (which is the letter t). Therefore, THIS cell equals T"

In a semi related matter, this article on the creation of QWERTY was really interesting:
http://www.strath.ac.uk/media/departments/economics/researchdiscussionpapers/2013/13-24FINAL.pdf

It looks like that layout image you posted does not really correspond to the text versions you wrote, and it looks like some letters are duplicated in the image.

Well spotted. ! should be J and O is repeated because I allowed for 34 finger positions but there are only 33 characters in my table. Excel produced a 0 to indicate a blank cell and I misinterpreted it as O. I must have written up the section of my post before I discovered that I'd left ! in the table by accident. The picture is more accurate of my intended design.

I haven't taken the time to figure out how to use your tool, but it's interesting that I can fix 3 of the top 4 cases by swapping keys 1 and 2 (T and E). I wonder if that would cause a rise in other bigrams. Swapping keys 13 and 14 may also help (D, L), or possibly hinder. I'm not sure where to go after that as I've only come up with an algorithm for placing characters according to frequency. I cannot think of a non-trial-and-error method to account for same finger units etc.

[lydell]

How did you create table one? This is how I understand it:

• Create a column of all ranks, sorted from the best to the worst.
• Create a table with two columns, key and corresponding keystrokes, sorted by keystrokes from the most to the least.
• Move column of ranks into the table, as the first column.

If the above is right, and I get the spreadsheet logic right, do you mean that it works like this: For the highest rated key, choose the most frequent letter, for the next highest key, choose the next most frequent letter, and so on? That’s actually what I tried to do initially, but I quickly realized that this led to many same finger usages and missed roll opportunities (both of which were very important to me).

[Snarfangel]

In a semi related matter, this article on the creation of QWERTY was really interesting:
http://www.strath.ac.uk/media/departments/economics/researchdiscussionpapers/2013/13-24FINAL.pdf

That *is* interesting. Fun read. It would be interesting to see the set of keyboards with "Sholes Optimality" (or "Jam minimization," or whatever you want to call it), and then see which one scores best with modern criteria -- common keys on home row, rolls preferred, that kind of stuff. It could tell us the keyboard we *might* have had the luck to use rather than QWERTY.



It's interesting, too, that he put all of the vowels on one hand. *So* curious about that now.  )

[Zustiur]

How did you create table one? This is how I understand it:

• Create a column of all ranks, sorted from the best to the worst.
• Create a table with two columns, key and corresponding keystrokes, sorted by keystrokes from the most to the least.
• Move column of ranks into the table, as the first column.

If the above is right, and I get the spreadsheet logic right, do you mean that it works like this: For the highest rated key, choose the most frequent letter, for the next highest key, choose the next most frequent letter, and so on? That’s actually what I tried to do initially, but I quickly realized that this led to many same finger usages and missed roll opportunities (both of which were very important to me).

Close but not quite.
Table 2 and table 3 were both shaped like a keyboard alpha section. 3 rows of 12 keys. I tried to show it in forum code in the first post but it doesn't display well.
I can post pictures of the spreadsheet if people want me to.

[lydell]

I _did_ understand table 2 and 3, I was asking about how you created table 1, and how the key placement worked at a higher level

This is how I understand table 1 one was created (note that this says nothing about table 2 and 3):

• Create a column of all ranks, sorted from the best to the worst.
• Create a table with two columns, key and corresponding keystrokes, sorted by keystrokes from the most to the least.
• Move column of ranks into the table, as the first column.

This is how I understand the key placement at a higher level:

do you mean that it works like this: For the highest rated key, choose the most frequent letter, for the next highest key, choose the next most frequent letter, and so on?

[Zustiur]

Oh sorry. Misreading on my part there.
Yes exactly.
In practice table 1 is a table with two columns. Rank and key. Yes.
It was built from the data on Wikipedia so it also contains a column for letter/symbol frequency as a number out of 1000.

The discrepancy detected in my diagrams earlier was because I forgot to remove ! from the list Aug first.