I generally pick from a color palette I know I'll like so it makes it easier to match later. So like if I buy a blue keycap set and some purple caps, I know those will go together. If I have a Valentine's set, those will play into it well....
But now that I think about it, I've put a lot of effort into some of my builds and I was forced to use what was available with other builds. For example, I'm kind of stuck with the caps I have on my Kingsaver. But I've customized my Zero pretty well. And I've put some thought into my eventual Leeku 1800 build.
I agree though, I spend *a lot* of time thinking up color schemes. Or wondering if I should just get a base neutral color like you said and then just swap in a bunch of fun caps.