The basic idea of something being "fatiguing" is that it may seem to sound okay at first, but when listening to it for a long time, something does not seem quite right about it. So that after a while, one does not like the sound anymore.
Edited to add:
As far as objective criteria goes, the feeling of fatiguing is subjective, though it can be brought on by small amounts of unpleasant sounding distortion that can be objectively measured. It can also be brought on by frequency response issues (that also can be objectively measured), where some frequencies are overemphasized, which may be exciting at first, but become unpleasant over time. I have found that to be the case with speakers where they overemphasize upper bass (often in an attempt to hide the fact that the speakers don't produce actual deep bass), which can make them sound "powerful" at first, but its wrongness becomes more obvious with extended listening. Some people, though, like that kind of boominess, and so it is not fatiguing to those who like it. The same feeling of fatiguing can be brought on by excess treble, where, at first, it may seem to be added clarity (as one hears the overemphasized frequencies more clearly), but over time may seem shrill and unpleasant. Also, of course, for you to notice that the frequencies are overemphasized, you have to listen to music that contains those frequencies or you will not hear the problem. So you may not hear it at first because of the particular music that you first hear on the speakers.
In other words, sometimes the wrongness of the sound is not appreciated at first, and only becomes apparent with extended listening.
Sometimes, though, the word is applied inappropriately (just like every other word).
Basically, a non-fatiguing speaker (or whatever) is enjoyable to listen to over an extended period of time, so that the flaws of the thing are not particularly annoying. Notice, though, what annoys one person may not annoy another, so the concept fundamentally involves a subjective reaction to whatever it is that is heard. With that in mind, something that accurately reproduces whatever is fed into it might be fatiguing to some person, who wishes to have some pleasant distortion added to the sound (which explains why some people prefer amplifiers and other things with high levels of distortion, over amplifiers and sources that lack that distortion), or to have some frequencies removed from the sound. An electronics technician told me of a customer who brought in a pair of speakers and wanted them to disconnect the tweeters on their speakers (I think they were 3-way speakers, but I don't remember). After explaining to the customer what that would mean, the customer said he wanted the tweeters disconnected. So they disconnected the tweeters. Since the customer never came back to have that changed, it would appear that he did not want to hear treble and was happy to have it removed from everything he listened to.
