There's the rub. I intend to finally recap my speakers this weekend - much simpler crossover than yours. I may try to push the lpad/crossover assembly out of the front baffle and will let you know, but I will not apply very much force at all. A fellow aker was able to pop his out relatively easily, which is encouraging.
Another alternative for you is to simplify your crossover significantly. It would be a fixed system, without all of the options for turning on/of or attenuating various drivers. Really, all you need to accomplish the "wide open" circuit are the 10uf capacitor, the 10 ohm resistor at the bottom of the crossover diagram and the 5 ohm resistor. Everything else can be straight wired, eliminating the potentiometer, switches and 10 ohm resistors associated with the switches.