There are a couple of things that stand in the way of making things smaller. Placement of the USB port, reset switch (make this a set of pads you jump, instead of a switch), the size of the package of the controller, say TQFP vs QFN, and the size/spacing of your solder pads for your wiring. The 2.54 pitch through-hole spacing on the teensy and the like just takes up... well... space... but if you're going to solder SMD bits anyway, maybe go with 1.27 pitch, or make pads to solder to, and use both sides of the board (instead of through-hole). I don't know what everyone is using these days, but an ATMEGA32U2 would be smaller and still give you all the connections you need - even better if you have what it takes to solder the 5mm QFN package vs the 9mm TQFP one. I'm sure you can make something quite small if you design it yourself and make it specific to your requirements. At that size, it won't cost much to make either.