I agree that treating it like a refoam is the way to go. The magnet is a very simple piece of the speaker. It just sits there, and isn't directly attached to anything but the speaker frame. It's the voice coil (basically an electromagnet) that is the complicated part. I would pass on using shims though, as that usually requires you to butcher the dust cap, and very likely never get it looking as good ever again. 30hz test tone method should be plenty if you go slow.
Does any part of the speaker (especially the voice coil) look like it sustained any damage as the magnet was detaching?