Keyboard Firmwares: a survey of controllers and converters

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 04:02

Which firmware to use?

In this thread I begin take you through a guided tour of the prevailing keyboard firmwares on the market today. I focus on open source efforts with some exceptions. I discuss pros and cons, uses, supported targets, architectural decisions, and so on. These are vaguely sorted in order of oldest to youngest and simplest to most complex. Let's jump right in!

For starters it's helpful to lay out some definitions. Feel free to skip this, I'm laying it out for lack of ambiguity and for newcomers.
Spoiler:

firmware: real time code running on a microcontroller that, in our case, is responsible for some aspect of transmitting key press signals from the bare hardware to the host

wire protocol: a specification for transmitting data down a physical wire. A field of study in itself. RS232/RS422 serial, Apple Desktop Bus (ADB), PC/XT, PS2, and USB are all examples of wire protocols (or in the case of USB, contain numerous wire protocol elements).

converter: a firmware that translates one wire protocol into a different wire protocol. Example: A PS/2 to USB converter.

controller: a firmware that directly senses the state of the physical keys, possibly emitting a signal along a particular wire protocol.
First a brief section on microcontroller choices. Maybe skip this section if you're already familiar with AVR 8 bit and ARM Cortex M0 MCUs.
Spoiler:

I prefer the ATmega32U4 since I'm gamiliar with it. The ATmega32U4 is plenty capable of just about any keyboard sensing task running at 16 MHz. However, it's not quite enough to drive interesting backlighting. We'll get back to that.

For many keyboards, we need only sense up to 128 key positions, which we can sense with an 8x16 matrix. So, ideally, for a device to host a controller, we're looking for a microcontroller with at least 24 inputs. The ATmega32U4 offers these inputs:

PB7..PB0: 8
PC7..PC6: 2
PD7..PD0: 8
PE6, PE2: 2
PF7..PF4: 4
PF1, PF0: 2

Total 26

A datasheet for the ATmega32U4 is available at http://ww1.microchip.com/downloads/en/D ... asheet.pdf.

The ATmega32U4 features prominently in the Pro Micro ($19.95; https://www.sparkfun.com/products/12640) and the Teensy 2.0 ($16; https://www.pjrc.com/store/teensy.html). The former has 18 I/O pins available, or 20 if you are willing to desolder the RX and TX LEDs. It is better suited to converter projects. (Schematic: https://cdn.sparkfun.com/datasheets/Dev ... o_v13b.pdf) The latter exposes all the pins and offers 25 IOs (though you may want to skip D6 or desolder the LED) and is well suited to controller projects. (Schematic: https://www.pjrc.com/teensy/schematic2.gif)

As of this writing ATmega32U4 based Pro Micro and Teensy 2.0 knockoffs can be had on aliexpress.com for under $5 apiece, handy if you have a pile of boards you'd like to convert.

For backlighting applications something with a bit more horsepower may be in order. For this the community is adoptin ARM Cortex M0 MCUs. Some basic information about this architecture is available at https://developer.arm.com/ip-products/p ... /cortex-m0.
A little bit of controller theory follows. Skip this if you're familiar with the software architecture of a controller.
Spoiler:

The fundamental problem that a controller solves is that of reading physical key state and outputting it via wire protocol. (Diagrams created using asciiflow.com.)

Code: Select all

+----------+     +----------+     +--------+
| Keyboard +---->+ Firmware +---->+Wire    |
| Matrix   |     | Loop     |     |Protocol|
+----------+     +----------+     +--------+
That's the basic architecture. But what's inside that black box, the "firmware loop"? Let's have a look at a possible firmware for a very simple keyboard with no macros, mo customization, no layers, and NKRO (no ghosting).

After doing some initialization at boot time, the firmware runs in an infinite loop that we can model with this pseudocode. This example is optimized for readability and little else.

Code: Select all

const int ROWS = 8;
const int COLUMNS = 16;
int[ROWS][COLUMNS] keycodes;              // 2D array containing key codes for each key.
bool read(int row, int col);              // Function to query a particular switch state.
void emit(int keycode);                   // Function to transmit a key code.

void main_loop() {
  while 1 {                               // Run forever
    for (int r = 0; r < ROWS; r++) {      // For each row
      for (int c = 0; c < COLUMNS; c++) { // For each column
        bool state = read(r, c);          // Read switch state at this position
        if (state) {                      // If this particular switch is set:
          keycode = keycodes[r][c];       // Get the key code for this switch, and
          emit(keycode);                  // Send the key code down the wire.
        }
      }
    }
  }
}
A quick calculation tells us how fast we must run. Assuming contact based switches, we perform sensing by injecting a test current and measuring to see if the current passes through the switch. It is required to wait for the switch to "debounce", or settle, for 1-5 ms.

128 key positions, times
5 milliseconds debounce time per read, yields a scan rate of
1.5625 hertz

Clearly that's unusable, so, can we read an entire row at a time? In this case that gives a 16x speedup, or, 25 Hz. Reducing the debounce time to 1ms yields another 5x speedup, to 125 Hz. That's faster than most monitor frame rates and is good enough for most purposes.
What are the things that I look for when evaluating a keyboard firmware?

Well, I'm a hobbyist first and foremost, but I always have mass market production on the back of my mind. So I'm looking for a few things:

- How hard is it to define a custom layout?
- How active is the community for this firmware - and how likely am I to get a response if I have a problem?
- How much does it cost? (Ideally: free.)
- How annoying or expensive is it to procure hardware that can run this firmware?
- How is ghost key blocking supported? (Watch for a future post on ghosting based on some notes I have.)
- How are macros, layers, and remaps supported?
- How much configuration can be done on the keyboard, versus "interactively" in a text editor or using a formal GUI or CLI tool?
- Are exotic hardware features like backlighting, a built-in screen, a rotary encoder, a split, or a solenoid supported?
- Are exotic software features like complex layers (e.g. SpaceFn, http://geekhack.org/index.php?topic=51069.0), settings banks, and (when applicable) per-switch actuation supported?
- How "familiar" is the software in terms of architecture and layout? How tough is it to tease out something like the psuodocode I wrote above?
- What does the workflow look like for a Linux or Mac user?
- Intangibly, how much do I like it?

Empirically, my goals are to support two keyboards. One is the Northgate Omnikey with NKRO, my absolute favorite keyboard hacker board. IT's easy to find, built like a tank, has full NKRO, the matrix is actually documented, it has fun hidden DIP switches, and the solder pads are huge and an absolute delight to work with. The other is the XMIT Hall Effect keyboard, which I'm working to make modern with a newer firmware for previous customers.

Let's begin with the classics: Soarer's Converter and Soarer's Controller.

Converter: viewtopic.php?t=2510
Controller: viewtopic.php?t=6767
WYSE Converter: viewtopic.php?t=7424

These are the controllers and converters I cut my teeth on. They're easy to use and they work. They're also, sadly, closed source. I like the tools that come with them but they're pretty basic. Soarer's Converter is great for converting AT to USB in a pinch.

I ran out of time, so, that's all for now! In the coming little while I'll follow up with investigations for:

xwhatsit
CommonSense
Kaleidoscope by keyboard.io
Kiibohd by Input.Club
TMK by Hasu
QMK
Das Keyboard Q

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 04:02

Reserved #1.

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 04:02

Reserved #2.

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 04:03

Go look at the original post! Reserved #3.

User avatar
Muirium
µ

23 Nov 2019, 09:05

Nice idea to write these up as a guide. For newcomers sake, I’d highlight how these actually appear in use, too. Even as a veteran keyboard remapper, I happen to prefer a GUI to a text tool: so my faves are Unimap (formerly gnarly TMK now neatly hidden behind the scenes) and the very visual friendly Xwhatsit.

Speaking of Soarer specifically, his controller / converter is a hybrid of open and closed source. He said he would likely open source the whole thing before he disappeared (and there is a reverse engineered source that someone made and gave me, but refused to open source without Soarer’s still absent personal permission). The current state of affairs is the converter hex file itself is (assumed) closed source while the tools users use (scas and scwr etc.) have always been open source, and I even compiled them for another architecture once myself.

You mention the user experience on the Mac. Now that Apple’s killed 32 bit, Soarer’s tools are dead. I need to recompile them for Intel 64 bit sometime, as I did for PowerPC back in the day. Depending on dependencies, I don’t know if this will work, though. PowerPC was a bit of a fight, and never worked on anything but Leopard.

Also broken in the 32-bitpocalypse: PJRC's Teensy Loader. Fortunately, there's an altogether better tool nowadays for all that: the very helpful QMK Toolbox. My TMK workflow is entirely source code and indeed terminal free thanks to Unimap giving me a GUI front end and the resulting .hex file, which then goes through QMK Toolbox to the Teensy.

User avatar
snacksthecat
✶✶✶✶

23 Nov 2019, 14:49

Your explanation and code snippet of the scan logic is so succinct and easily digested.

Looking at the TMK code initially intimidated me but I had a moment of clarity once I realized that it's just been architected to do a lot of different things. Once I got past that, working with it and tweaking to my own needs suddenly became a doable thing!

Kiibohd always interested me as well but I never dipped my toe into that pool.

I look forward to seeing where you go with this post. You, in particular, have a unique perspective on the topic having worked on a commercially available product.

User avatar
XMIT
[ XMIT ]

23 Nov 2019, 15:18

Now that Apple’s killed 32 bit, Soarer’s tools are dead.
Ugh. This further validates my refusal to "upgrade" from 10.14 while I have important projects afoot.

User avatar
Muirium
µ

23 Nov 2019, 19:37

Smart move. Catalina's a right pain in the jacksie in several respects. But you can expect no remorse from Cupertino! 32 bit is as dead as Classic.

So, I tried to build Soarer's tools from source. Their source code is in the tools folder within Soarer's zip file. After figuring out just where exactly Apple moved the SDKs…

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include

…and replacing i386 with x86_64 throughout, make is still barfing volumes. Apparently there is more to this 32 to 64 bit lark than I understand! See if a bona fide coder like yourself can pull it off. We could all use scas and scwr and friends in 64 bit.

User avatar
OldIsNew

23 Nov 2019, 20:27

Very interested in seeing more! Thanks for working on this, very helpful idea to put a run down on things in one place.

User avatar
XMIT
[ XMIT ]

24 Nov 2019, 02:04

Apparently there is more to this 32 to 64 bit lark than I understand!
I'll see if I can take a look though this is pretty low on my list. Arch changes tend to bring out integer size bugs. Anyone tried building these on Linux x86_64? Getting it up there would be easier.

User avatar
SDS604

27 Nov 2019, 09:53

I am surprised nobody has mentioned Jorn and his wonderful viewtopic.php?t=17388 BL-USB Bluetooth Model M Controller Conversions. This solves the issue right at the root instead of trying to adapt.

User avatar
Muirium
µ

27 Nov 2019, 10:39

Those are just a product, though, rather than a project. And only work for one model line of keyboards: IBM Model M.

The controllers (yes we are also talking “root” here) worth discussing are general purpose and ideally open hardware and open source.

User avatar
vvp

27 Nov 2019, 21:47

XMIT wrote:
23 Nov 2019, 04:02
Which firmware to use?
Chris' firmware: https://github.com/chrisandreae/keyboard-firmware
or my branch of it: https://github.com/hercek/keyboard-firmware

Features:
  • Almost no community. Chris did respond to my questions when I had any. I can respond to questions about my branch.
  • Uses LUFA or vusb (so MCUs which are supoorted by it, e.g. atmega, atxmega). You need to add your backend. Provided backends are for ergodox, Chris' kinesis advantage custom controller, Katy custom controller. You need to port it for other HW.
  • Custom layout can be modified by an GUI app or on the fly using Program key (as Kinesiss Advantage).
  • Open source.
  • HW is hard to produce (if you consider Ergodox building hard).
  • Remaps over GUI or through Program key. Macros over GUI or through Program key. Programs over GUI only. Programs are a small interpreted pieces of code in a C-like language which you compile to a p-code and load to firmware and associated with a trigger using an GUI application. All these things do not need firmware re-flash.
  • Mouse keys support. No unlimited sticky keys or dual function key support. Two layers (normal and keypad) or three layers in my branch for K84CS (normal, keypad, function).
  • Maximum number of macro triggers and macro storage (taken by all macro definitions together), and program storage limited by a compile time constant. At most 6 programs are possible. Storage locations are eeprom, spi-eeprom, i2c-eeprom, flash (not sure this one works well on the fly).
  • GUI can do everything except definition of macro sequences themselves. Program key (on the fly) definition can do everything except program definitions and trigger assignment to programs.
  • No backlight support. LCD support but only for Katy.
  • No pseudocode compiler. There is compiler for custom very slashed down C-like program definition language (e.g. it support emitting keys, checking key presses, precise delay, mouse keys). A new HW suport must be added by implementing an interface for the HW control in C.
  • Workflow: mostly as Kinesis Advantage with perks (macros, GUI, ...).
  • The GUI application is in QT and compiles at least for Windows and Linux. It probably will compile on Mac ... with some tweaking. One needs to instal also libusb driver for Windows.
  • 6KRO only. I did not see the allure of NKRO but it can be added by a C coder. The matrix scanning is more or less NKRO. Only the protocol is the bios keyboard (6KRO).
  • Some special features like LCD, on-the-fly debounce length configuration and mouse keys speed configuration in my branch.
  • Probably not useful for people who are clueless about C if they are not going to use it with the exactly supported keyboards.
  • You probably never heard about it so you do not have an opinion .... yet :)

User avatar
XMIT
[ XMIT ]

27 Nov 2019, 22:39

Oh cool, I'll definitely take a look! :D

User avatar
XMIT
[ XMIT ]

27 Nov 2019, 22:42

SDS604 wrote:
27 Nov 2019, 09:53
I am surprised nobody has mentioned Jorn and his wonderful viewtopic.php?t=17388 BL-USB Bluetooth Model M Controller Conversions. This solves the issue right at the root instead of trying to adapt.
Oh, this project happened while I took some time away from keyboards. I'll have a look at this one too.

If anyone else has firmwares to recommend please continue to update the list. Though the Omnikey and the XMIT Hall Effect are my top two development target boards, it's fair to say that the venerable Model M is #3.

MMcM

28 Nov 2019, 19:11

For the future, I think Circuit Python may be an interesting place to put bets. Adafruit is making boards close to the right form factor and price. And she has people helping to make it a usable ecosystem.

For keyboards, there is someone on Tindie selling a macropad. The scan loop and other software beyond the standard libraries looks to still be pretty primitive. But, in the end, I think this may all prove more approachable.

PiKeeb, though, seems like it's too much and so ends up being unsure what it's supposed to be.

Post Reply

Return to “Keyboards”