The absolute polarity, commonly called "acoustic polarity" and, mistakenly, as "absolute phase", also matters. This is the direction of compression or rarefaction wavefronts. In short, does a positive signal move the speaker forward or backward. The reason this matters is that our ears were optimized by evolution to prefer compression before rarefaction.
This difference occurs because of the
Wood Effect. See
The Wood Effect: Unaccounted Contributor to Error and Confusion in Acoustics and Audio by Clark Johnsen (1988). You can do a search on the Internet for further reading.
Somewhat simplified, the issue is a difference in how the ear and brain process the different waveform types:
Compression Wave (0 —> Vmax —> 0)
versus
Rarefaction Wave (0 —> Vmin —> 0)
In this case Vmax is the peak for the positive half of the waveform and Vmin is the peak for the negative half of the waveform. They are used to convey the different halves of the waveform in an AC cycle.
Compression
before rarefaction occurs in instruments and real-world sound;
the ear/brain prefers this relationship. An additional complication is that the two waveform halves are rarely symmetric for real-world sound, and the speaker may exacerbate this asymmetry, distorting the sound.
You'll find that the following situations can apply:
(a) playback equipment reverses polarity (preamplifier or DAC) compared to recording
or (b) the recording's polarity was reversed in recording, playback equipment maintains it
or (c) both equipment and recording polarity are reversed
or (d) both equipment and recording polarity match
Some equipment contains a polarity reversal switch to restores proper phase. This is different than a switch to swap the polarity of only one speaker to make both speakers have the same polarity (i.e. the +/- at the amplifier matches that at the speaker), but not absolute polarity (i.e. both speakers move in the same direction, but do not prefer compression before rarefaction). So the polarity may be mostly correct or mostly incorrect depending upon your equipment, but even when you have the right phase for the equipment the individual recording may or may not be in absolute polarity with respect to playback equipment.
In summary, configure your speaker polarity for the majority of recordings and switch it for the minority, and rejoice in discovering new depths to sound.