It's probably a driver issue. When your keyboard is just plugged in it's seen as a keyboard, using the OS keyboard driver. No problem. When you click the set bootloader button, the keyboard "disappears" from the OS and the keyboard controller appears. If the OS doesn't have a driver for the controller (AtMega 32u2, 32u4, etc), nothing happens, as you've seen.
I don't know anything about your board but you need to research the controller and find/install a driver for it.
And...
this might help.