OK I had some ideas:
First, you could make the whole thing as big as the inner diameter of a lego piece, that way a brick fits further down(but the walls are really thin at that point):
Show Image
If those walls are too thin, you could get rid of them like this!
Show Image
personally i would say keep the walls.
mainly because in the past i bought a cheep key cap that didn't have the walls on the stem,
and it seemed to wear out just from taking it on and off the key switch itself. so now its fairly UN-usable.
but depending on the materials it could be different.
who knows maybe i'm just rambling.