I would not recommend a 4-pin RGB hand wired project, this is because it would be crazy to handle. If you want RGB and handwired, then use those WS2812B strips, each of those LEDs has an incorporated controller already to handle the the colors.
You could have all the LEDs connected to 4 pins on the Teensy, this will however allow you top have only one color for all the LEDs as there is no way to address the LEDs individually.
Products using 4 pin RGB LEDs handle the addressing and setting the value for each RGB channel using,
1. Individual controllers for each LED, like the WS2811 and then just chain the output from one LED controller to the next one's input. So only one pin goes from main MCU to the first LED controller.
2. Single or multiple dedicated LED controllers, depending on LEDs supported by controller and number of LEDs in use, eg IS31FL3732 supports upto 32 RGB and connecting each channel from the RGB in a matrix to the controller. The controller can communicate with the main MCU via I2C/serial/etc depending on IC.