Not knowing much about your listening room acoustics, treatments and/or eq, could it be the unused set of speakers effecting the acoustics??? If you are playing both sets of speakers at the same time then there could be some other issues to look at with something like Room EQ aka REW...