AR used some gummy putty type material to seal the drivers as a gasket. It can hold onto the drivers pretty firmly. You need to pry some of it out from the edges and try to get something in there to help pry it lose. Be careful not to damage the wood at the edge on the baffle around it of course. If the woofer comes out easier, as RxDx says, go in there to push from underneath. it will eventually give.
If you haven't got the drivers out yet, how did you test it?