Cool, I'm glad someone else is interested in this! Here is the repo with source code and docs I put together:
https://github.com/p-codes25/avant-programmerNow that I look at it again, the formatting isn't quite right in the KBD file layout page -- it should use 8-space tabs but it's showing as 4 now... there might be other glitches like that, because I'm pretty new to GitHub. Also, I uploaded all this 4 months ago but the page still doesn't show up in a Google search, so maybe I need to make a Gist page or something else to make it searchable?
As you'll see in those docs pages, the layout on the memory chip, the *.kbd file format and the on-the-wire protocol are all similar and related but not quite the same. So a lot of the code in my utility deals with converting between those formats. To edit the *.kbd files, I used CVT's 'new' Win2K/XP GUI (Avant.exe, which works fine through Win7, except for the programming upload/download part, of course).
What kind of special key codes do you want to generate with the top F keys? I'm not sure exactly what the keyboard might be capable of, beyond what's supported in the old CVT GUI software. Unfortunately the 82C51 chip seems to be locked, at least on my Avants, so I wasn't able to read the firmware to see what goodies might be lurking in there...
But as far as the existing key remapping and macro protocols, they seem to only be capable of sending keystrokes that any of the stock keyboard keys can generate -- each remap or macro keystroke entry gives a 'key slot' as I call it, which indicates a press of a key position on the stock keyboard layout. I don't know if it can generate arbitrary scan codes that didn't exist in the stock layout though, if that's what you need.
I haven't worked with drivers on Linux, but I could maybe help in terms of understanding the i8042 hardware and how to work with it for programming these keyboards using their existing protocol. Otherwise, or if you wanted to program one through a USB port, it might take coming up with new firmware and a new controller chip in the keyboard itself. That would be a fun project, but I haven't looked into it.
Let me know if you have more questions and I'll see what I can do...