Both USB VCC are shared, so the lowest resistance connection is the one that will feed everything, I smell that its going way over the 500mA limit and the port gets disconnected.
Or maybe the fact that the USB lines jump over vias a couple of times and run all across the board without a ground plane.
Also, are the USB D+ and D- from both micros really connected in the W1 and W2 jumpers?!