Buy some Clears and swap out both the stems AND springs.
A stronger spring on its own will tend to minimise the already small tactile bump of Browns.
I would actually recommend going all the way to 62g ErgoClears by buying Clears AND 62g springs and using the stems from the Clears with the 62g springs in the housing of the Browns. Best feeling switch IMHO. You can also use a stock Clear spring in the space bar switch, though, to help even out the feeling.