Absolute phase is important, but not everybody hears the difference. Imagine a kick drum, with correct abs. phase it will increase sound pressure on the attack, otherwise it will generate negative, 'sucking' sound pressure resulting in unnatural sound. The difference is definitely not in-your-face. In any case, always have left and right speakers or headphone coils wired with the same polarity.
Ideally, you have a phase switch somewhere in your system to compensate source material that is recorded with wrong absolute phase. For example, with any new CD I listen to, I check whether 0 or 180 degree phase sounds better, and then keep this setting for this recording. I can achieve that with a remote controlled DAC.
What's more important than any theory is that it sounds right to you. Use what sounds better to you!