Apologies if I missed something, but I'm really confused about the initial question. The bigger challenge with multiple amplifiers and one set of speakers would be to do the speaker switching between amps rather than the RCA input switching. I don't think it's a good idea to wire the amps in parallel to the speakers, even if only one amp is on at a time, you'd really want a speaker switch to control which amp drives the speakers (sorry if you discussed this and I missed it).
For the RCA inputs, it seems like a better solution would be an "active RCA 1 to 8 (or whatever) splitter" that would allow you to drive all the amp inputs in parallel. Then you'd always have the same signal at all amps and you would only need the speaker switch to select which amp is driving the speakers at any given time (instead of having to do a second step of manually redirecting the RCA input). You could use RCA "Y" splitter cables to drive 2 or 3 amps, but not for the number of amps you want to grow to. An active splitter seems like a solution that is far easier to use by essentially eliminating any need to do RCA switching.
Of course, If you needed to be able to switch sources, from CD to WiiM for example, for all the amps, you could put an RCA switch
before the active splitter to change sources easily, but you wouldn't need to do a second set of switching to dictate which amp you want the RCA signal to be routed to.
At a glance, it looks like something like the device could feed the RCA to up to 8 amps in parallel. I haven't looked to see if there are higher quality versions of this type of device (but the Amazon reviews seem to be quite good for those using them in audio systems).
Amazon.com: DTECH Powered 8 Way 3 RCA Splitter Box 1 in 8 Out Composite Video Audio Distribution Duplicator with Power Adapter Compatible with TV Projector Receiver Gaming Console Video Capture Card : Electronics
www.amazon.com