More
Really cheapest?
* Switches and keycaps from junkyard/eBay/flea market find(s). The 1.5u keys and 2u keys might need to be taken from several keyboards. You might get the capacitor and resistors as well if you are lucky.
* Use PCB-mounted switches and through-hole diodes integrated into the switches => You don't need a plate. With two pins and four solder joints, every switch will be very sturdy.
* Build the case yourself from whatever you have around. There are holes through the PCB that could be put to use for standoffs for holding the case together. Or don't build a case at all - just cover the sensitive bits with electrical tape or whatever ...
* Skip the TRRS and USB jacks and connect directly to PCB and Teensy 2.0 respectively. If you don't solder on a separate USB jack, some PCB revisions may require you to run an additional wire between the Teensy's +5V out and the PCB or otherwise the other half will not get power.
* If you don't want them, skip LEDs as well, and the resistors that go to them.
But is this plan tp approved?