Even if you sort out the grounding issues and have no hum, you are still adding more connectors and switches to the circuit. That won't really make much of a difference with a line-level signal, but with the tiny signal that comes off of a cartridge, you might find the sonic degradation to be audible. An AV switcher is made to handle line-level signals, not the weak signal of a TT cartridge, which is quite sensitive to capacitance, etc... of what it flows through. Your AV switcher will "work", but is likely to noticeably reduce sound quality.
Think of it this way: If someone is carrying a load of 100 bricks (=line-level signal) and you lift one off, they may not notice the difference; it's only 1 percent. But if they are only carrying two bricks (weak phono signal) and you remove one brick, it's a 50 percent change in their load, very noticeable! With a small signal, the less there is in the path, the better.
With a Y-adaptor, the other TT's wiring and cartridge are being put into the circuit, even though the other TT is off. Given the TINY signal coming from a TT, it's definitely a less-than-desirable solution.
Between those two options, the switcher might be the lesser of the two evils...
OR you could just plug-and-unplug each time. The risk to that, of course, is that you'll wear out or break the connectors much faster due to all the physical movement. But you'll get the best sound that way.
But the BEST way would be to have two phono preamps, or a preamp with more than one phono input. No losses due to additions to the wires carrying those tiny signals, plus no wear-and-tear from repeated plugging and unplugging, plus convenient flexibility over what you listen to. The only drawback is that it might cost more.