Another vote for the audio selector box.
I have a 3-way selector box that I bought at Radio Shack in the '90s and it works perfectly. I installed it between the tables and my preamp, and connected all of the ground wires to the same terminal on the preamp. No loss in fidelity that I can hear, and no hum issues either.
Audio-only selectors are somewhat rare these days, but AV selectors are easy to find and should work just as effectively.