One thing you could do is to split the rows in "half" to get 10 rows. Then you'd only need half as many columns, using each column twice, once for each of the splits. That way you'd only need 21 instead of 26 pins for the matrix (you could probably do away with a few more mashing things together further) and get away with a regular Teensy rather than the plusplus.
It's common practice to connect LEDs the other way around, with the cathode towards the controlling pin. It doesn't matter much with a Teensy, but some chips are better at sinking than sourcing current and people just tend to do it that way. It may also be handy to connect all LEDs to pins that are connected to the same timer. For example yours are connected to pins with OC1A OC1B and OC2A if you move the one on PB4 to PB7 you'd only have OC1* pins. That way you can control the brightness with PWM using only one timer, which will be a bit cleaner to set up, and you'd not occupy the extra timer (which you aren't short on though).
Also, beef up on the traces, there's no reason to keep them skinny. Increase the spacing since you've got the room anyway. With tat in min, still tighten them up. Use more 45° traces and cut more corners. If you route some of the matrix lines from the controller underneath it, and move some more of them to the bottom layer you could slim down the board a bit (you may be physically constrained by the case to put the Teensy exactly there though). Green and red are allowed to overlap, that's the whole point =D
And no, you don't necessarily need a ground plane. Topres have them, but they do magic capacitive stuff. I have still to see a commercial cherry board that use one.
Are those half circle things cutouts for a ziptie or something? You probably want to make them regular round holes. Funky shaped holes usually cost extra.
If you put the matrix diodes on top as well the bottom of the board will be cleaner. They become a bit inaccessible for debugging if you use a plate though. Through hole diodes is a middle way. You can measure on them, but still not easily swap them out if broken.