I am in a similar situation as your's, too many sources for only one receiver.
I solved this problem by connecting many sources to each receiver, and I use the Tape monitor 2 of both receiver to interconnect them. I can hear sources from receiver 2 on receiver 1 by switching tape 2 monitor on receiver 1, and vice-versa. It works very well.
For your speaker problem, there is a simple BUT NOT VERY SAFE solution. In your gear, there is a speaker switch. Signal is going to headphone at all time but the speaker switch must be depressed to hear sound from the speakers.
This switch is inserted between the amplifier and the speaker connector. You can connect all speaker output in parallel (all Right channel + together, and so on).
With this set-up, all you have to do is switch off the speaker from the receiver you don't want to hear, and switch on the speaker on the receiver you want to hear.
BUT NEVER FORGET TO SWITCH OFF THE SPEAKER SWITCH FROM THE RECEIVER YOU DON'T WANT TO HEAR. You can power on the receiver and use the phone output.
Take care!