The impedance mismatch will affect the crossover frequency and possibly, slightly, the output level. The change may not be audible, depending on the slope of the crossover. To be honest, I suspect your listening room will affect the sound more than the impedance change.