NerD60 - TMK Conversion / It works !!!

[DanielT]

After using for a long time NerD60 boards I discovered some small flaws in the firmware design and I would like to switch to TMK firmware as the controller an ATmega32U4 supports it and there is also support in TMK for these boards.
The big problem is that I don't have any clue on how to write a new bootloader on the ATmega32U4 chip.
So what I would need is a step by step guide on how to do this, what kind of ISP programmer works best, what software I need, how to connect and suff like this.
I know there are a lot of people who would love to make this conversion but without a decent guide nobody wants to brick a quite expensive board.
My intension is to have this thread as a reference for this. If I manage to gather enough information I will document my conversion. I really want a decent firmware on this otherwise very nice PCB, don't want to fire up my Win7 vm every time I want to make a change. Also I have noticed some problems with BIOS/GRUB and even on OSX (not very fond of the last one but it's my work laptop so I need a decent board)

UPDATE: This works and there is also a Wiki article for this wiki/Converting_NerD60_to_TMK

[scottc]

Possibly not related, but ever since I plugged my FaceW into a USB 3.0 Hub (Anker, from Amazon) it seems to be recognised in the BIOS.

I still prefer TMK to everything else though, being able to use standard tools like git/vim/etc on your keyboard layout is priceless.

[DanielT]

With faceU/W the powered hub solves the problem, but with NerD60 it doesn't make any difference, I get strange behaviour , after a fresh start some keys are not registered and it messes also with the Mac Book keyboard, I have to unplug the NerD to solve it.
And I really want TMK on the board, it's platform independent and has better support.
The only problem is that I really don't want to experiment, I know people have programmed controllers with an ISP so I want a small guide on how to do it

[DanielT]

Bump

[hasu]

To program bootloader you need a hardware programmer like Atmel AVRISP mkII or ArduinoISP. Just hook it up to pins of controller of your PCB and use software like avrdude or Atmel FLIP. Note that once you replace bootloader you will never be able to recover the original bootloader because they don't seem to provide bootloader source/binary.

http://www.atmel.com/ja/jp/tools/avrispmkii.aspx
https://www.arduino.cc/en/Tutorial/ArduinoISP

But I'm not sure you really need to change bootloader. I guess you may program your TMK firmware with NerD_downloader.exe. I don't know about NerdD PCB and its tools at all in fact, just downloaded tools and saw the files.

[DanielT]

Hmm, I will try and see if I can use the NerD_downloader.exe for new firmware. If I write a new bootloader what would be it? Where do I find it ? I really have no clue how this works.

[jou]

LUFA has several bootloader implementations: http://www.fourwalledcubicle.com/LUFA.php

As for hardware programmer, you can use a spare Teensy or Arduino with ArduinoISP (as hasu said) or if you want a dedicated one, USBasp clones can be had pretty cheaply.

[DanielT]

I've just read that you can use also a Raspberry Pi for this, it would be easy for me asa I already have 2 of them. I'll have to study the problem more. I also have a teensy
Thanks for all the info

[flabbergast]

You can use either teensy or raspberry pi, both can be used as AVR ISP programmers. Adafruit has a nice write-up on how to use a pi for programming atmel chips.

I've done my fair share of playing with LUFA, so if you need help with compiling a bootloader, just give me a shout. A nice thing about compiling one yourself is that you can program in a check for deciding when it should run (e.g. "if you're holding a particular key during power-up, the bootloader should activate"). But hasu's right, you can keep NERD bootloader and just use gon's exe to program the board (the only thing that might not work right away is "jump to bootloader" magic combo).

Also, depending on how gon's set up the fuses on the chip, you might be able to read the whole flash on the chip (including the bootloader) with avrdude, essentially giving you the option to revert any changes you make.

[DanielT]

I will try with the NerD stuff and see if it works. It has a magic key combo for bootloader mode so it should be ok. Th only crappy thing is that the software works only on windows. But having a better firmware should be a step up

[DanielT]

Small update, just tested writing the TMK Firmware with the NerD tool, it doesn't work. So I will have to bite the bullet and flash the thing.

[flabbergast]

Hm. Strange. How "does it not work"?

Here's some explanation: the 32u4 has 32k of flash; and the bootloader resides "at the end" (in a "bootloader section", whose size is configurable to 512b, 1k, 2k or 4k I think; via "fuses" - which are really just a couple of configuration bytes, which can be changed only using an ISP programmer). "Normal" firmware resides at the beginning (address 0).

The way 32u4s in keyboards are usually configured (again via fuses) is that on power-up or reset, the MCU starts executing code from the bootloader section. The bootloader then decides if it should continue running ("some condition") or jumps to address 0, thus relinquishing the control to the firmware. So, unless the bootloader has some tricky code inside (e.g. which checks whether the firmware contains a particular "magic byte" or something, to prevent running "unauthorised" firmware), the bootloader and the firmware are sort of independent - that's why I said it should be OK to just use gon's bootloader stuff and run TMK. However nobody (except gon) knows what's happening in his bootloader...

More about "some condition" in the bootloaders. For instance, the ATMEL DFU bootloader, and also teensy bootloader, check whether the MCU was powered up (then it jumps to firmware) or whether the MCU has been reset using the reset pin (in which case it continues running as a bootloader). The way I prefer to do it is to check whether some button has been pressed, and decide to continue or jump to firmware based on that (this is for instance how all the face* boards do it).

[DanielT]

Well, the NerD software works like this. You plug in your keyboard, you don't have to press a key or something to put it programming mode, you just run the .exe indicating the .hex and it starts to write the new firmware. You see the status LED flash and that's it.
I have created the .hex ran the .exe LED started to flash but it didn't write a new firmware with a new layout. I made sure it's a whole different layout so I can be sure it works or not.
It seems that the bootloader checks if the firmware is generated by the NerD GUI or not.

I know form the classic face* boards, you use a key combo and it goes in "programming" mode, with NerD board you don't have that. You have a key combo to force it in bootloader mode but again works only with the official firmware.
So the only way round I see is a new bootloader. This should work because xauser did it, and he wrote the TMK part for NerD boards, also with LED control and stuff.

[flabbergast]

Thanks for the info! (Not that I have a nerd board, but it is useful to know.)

So as you say - since the original bootloader doesn't have a way of triggering it outside of the official firmware, the only way to go is replacing it. (Again, let me know if you'd like some help with that - there are some choices and particulars involved which make it a tad too long to write a "guide" at this point.)

[scottc]

Didn't derzemel do something like this recently with his broken TX1800?

[DanielT]

Ok, I briging this back to life. I hate the default firmware is has a lot of bugs and after getting GH60 I've started to like TMK more and more.
I have a Teensy 2.0 that I will use as ISP programmer, have to get some cables in order to connect to the keyboard and very important I have to figure out how the hell do I write a bootloader, flabbergast I will start to bug you for this part if you have the time

[flabbergast]

No problem. I wrote some notes about this here - but not sure what level of knowledge/support you'd need

[mohitgarg]

Sorry if I missed this somewhere but doesn't the Nerd60 use the Atmel DFU bootloader?

[DanielT]

It has a custom bootloader, life would have been so easy if it were DFU.

@flabbergast: I'll do some reading first and after that I will come with my questions. I have really no clue about this subject but I want to make it work.

[mohitgarg]

Interesting, I wonder what bootloader it is using, what's the current method for programming it. (Sorry, I don't have a NerD board)

Edit, read the posts above and see how it is programmed. Daniel, did you try to flash the firmware by passing the filename via command-line. I would guess that GON is using some kind of serial communication to restart the device into bootloader, flash the firmware and then restart the device. I ran NerD_downloader.exe with a test hex file and the message read, "Waiting for bootloader", this led me to believe that a jump to bootloader command is sent via serial, as that method is employed by Arduino too.

[DanielT]

I did that already, it doesn't do anything. See a few posts above, I did all sorts of experiments none of them worked.
Now I'm going to bite the bullet and hope not to brick am expensive keyboard But I have some reading to do before doing something stupid.

[flabbergast]

If you have access to the programming pins of the chip (reset, miso, mosi, sck) then the only way to effectively brick the keyboard is to program a specific fuse (BOOTRST). In other words, you can always reprogram the firmware whatever the firmware contains. And unless GON enabled some security thingies, you should also be able to read the current firmware from the chip (so that you have a backup and can always go back to the current state).
{We can arrange an IRC session}

[DanielT]

From the PCB I have access to all the pins so this part is easy. I will use Teensy 2.0 as ISP programmer so first I have to get used to the tools I spoke to xauser the user who made the TMK branch for NerD60 and he told me that GON protected his firmware so you can't copy it so this is a one way street
I will first read a little and study the "art" of ISP programming. After I know and understand the tools I'm going to use I will know also what questions to ask, don't want to take too much of your time
I have a technical background and in University I used to program microcontrollers but that was like 13-14 years ago.

[mohitgarg]

It's quite easy. I'm using my Teensy 3.2 as an ISP programmer, but the basic should be the same. Just flash the Arduino as ISP program on to your Teensy using the Teensyduino. Then it's just a simple commandline using avrdude.

If you want I can share the compiled LUFA bootloaders for the Atmega32u4 as well.

[DanielT]

Well, at this moment I have Arduino ISP programmed on my Teensy 2.0 . I have to build some kind of a connector, I guess it will be just some wires to the PCB.
What puzzles me is the following: I write the new bootloader , that's easy (at least in theory ) , next will be to write the firmware, on GH60 I have a reset button I can push but on NerD60 there's no button, how do I write the firmware ? I write the HEX with the ISP just like I do with the bootloader ?

[DanielT]

Well, I'm one step closer I think my ISP is working, I was able to read the damn thing and it looks OK. Now I'm tired and it's also not a good day to try some more stuff but I'm pleased with my getto ISP programmer

Code: Select all

@liz:~/Software/Teensy/arduino-1.6.8$ avrdude -p m32u4 -c arduino -P /dev/ttyACM0 -U flash:r:/tmp/dump1.hex:i -F

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0xffffff (retrying)

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0xffffff (retrying)

Reading | ################################################## | 100% 0.00s

avrdude: Device signature = 0xffffff
avrdude: Yikes!  Invalid device signature.
avrdude: Expected signature for ATmega32U4 is 1E 95 87
avrdude: reading flash memory:

Reading | ################################################## | 100% 10.77s

avrdude: writing output file "/tmp/dump1.hex"

avrdude: safemode: Fuses OK (E:FF, H:FF, L:FF)

avrdude done.  Thank you.

[flabbergast]

This looks like the chip is indeed 'locked' by GON (normally the values that are read are not 0xFF all the time).

About programming - normally when you flash a firmware, the whole chip is erased. So if you flash (say) TMK using the programmer, it will erase any bootloader that's there. The normal course of things is
1) flash a bootloader
2) set the fuses
3) do not use the programmer any more, any subsequent flashing is done via the bootloader.
{You can of course ditch the bootloader entirely and use the programmer whenever you're flashing new firmware; but this is usually not very convenient.}

About entering the bootloader: normally the fuses are programmed such that the bootloader is entered *every time* the board is reset or powered up. The bootloader code then checks for "a condition", which if true, it keeps running and if not, it jumps to the actual firmware.

The "condition" could be various things; the most commonly used ones are:
1) what's just happened was a reset and not a power-up {this is used by the original ATMEL DFU bootloader for instance}, but this is not very good if you don't have a reset button
2) check status of some pin {in other words, check if some kind of button or jumper is 'pressed'}

Since you'll be flashing a bootloader anyway, you might as well get one which for instance checks whether you're pressing a particular key during power-up. That way you will always be able to get to the bootloader even if the firmware goes haywire. Of course you can also 'jump to bootloader' from TMK, if you assign 'BTLD' keycode to a key.

[scottc]

While I don't have a GON keyboard and probably never will, I have to say that I find your explanations both fascinating and very informative, Flabbergast. Thanks a lot for sharing all of your knowledge!

[DanielT]

Hmm, so if I will write the ATMEL DFU bootloader I will not be able to put it in bootloader mode because I don't have a button or something like that.
So I need a custom bootloader with a key/switch that will do what I need. That is more complicated amd I will need to study more, I don't want to break my board.

So GON locked his firmware, that's why I get that message with device ID 0xffffff and worked only with -F option to read the stuff from the atmega32u4.
xauser said there is no need to do something with the fuses.

[flabbergast]

There are fuses and lock bits. You may want to clear the lock bits, just so that you can read the firmware back and check what the fuses are. The point is that this (clearing the lock bit) can be done only after a device erase.

You can't break your board by writing a non-working firmware - you can always reprogram the board using the programmer.

With ATMEL DFU, you will be able to enter the bootloader in two ways:
1) through firmware (jump from TMK for instance)
2) by shorting the 'reset' pad to the 'ground' pad (which you have access to, because you were able to hook up the programmer

EDIT: I can compile a bootloader for you, or help you with doing that, if you want.