Well, the IO expander couldn't really be blamed for the case of the cable plugged into the right half but not the left and still having this problem, though... maybe it's the TRRS connector itself
The Trrs connector is ****..
Directly solder the wires for now.. and see if that helps..
if you bought the kit form MassDrop, the cable is ****, the connector sockets are also ****... I can't emphasize that enough...
Both my Egdx kits are directly soldered... the first trrs connector broke (intermittent) only a week in, and I've only inserted maybe <20 times..
Eventually, they all flaked out... So now It's all directly soldered to board.. no problems ever since..
I can't promise that this is "your issue as well".. but the TRRS had been such a headache for me..
Again, we need pictures to really help you...
Double check that you got all the diodes in right as well.. for whatever reason.. when I tried the board before putting in the diodes, it would bug-out..