I can help you with this... but I'd need to know more about what firmware you're using and how you're connecting the matrix to the Teensy.
My guess would be, you're using PD6 as a row or column select (i.e. what you pull to ground so one row or column of switches is "read", with internal pull-up resistors). The pull-up resistors will make the inputs read as high if there's switches open, and low if there are switches closed.
TMK firmware, as an example, will output a low to select, and a high-Z state (i.e. effectively disconnected) for the pins not selected.
When PD6 is in a high-Z state, it's still connected to the LED and resistor to ground, so unlike the other pins, it won't be truly "disconnected" when that pin is not the "selected" row/column, and could cause the voltage on the input pins reading rows or columns to be not 0V but low enough to be read as a logic low.
Just a theory, though... if you are using TMK, you could fix this in matrix.c and change it to output high instead of switch to high-Z state.
Or (not that I suggest you do this yet), you could burn out the LED with high current so it's disconnected

Also... did you kill the pads on F5 and E6 too, or are you already using those and really need D6?