IF you can find one (and can afford it when you do), there are selector boxes out there that allow you to select from multiple inputs and outputs, routing the signal from wherever to wherever. These were/are usually used in dealerships to demo various gear for different customers. That's why there aren't too many out there. But they make a great solution, if you can get one.
There are also smaller models. I have three decent JVC switchers that each have two inputs and four outputs to select from. Adcom also makes a nice one with six outputs. By "daisy-chaining" them (using the output from one as an input to another) you can build up a larger capacity, although remember that the more switches or connectors you put in the signal path, the more chance for sonic degradation.
As for switching speakers around using wires (not a switchbox), I agree that banana plugs (using those adaptors where needed) is the easiest route. You COULD also decide to replace your connectors with ones that accept bananas, and do away with the need for adaptors.