I believe these speaker frames are made from Aluminum. So, it's likely not rust, but something else? Probably oxidation, and some other staining, could be from Cigarettes, etc, giving that brownish look.
Before using any physical means, I would see if you could somehow acquire a liquid Aluminum Cleaner if possible? In other words, first try a liquid, but as you guessed it, one will need to be careful, Apply any fluids to an MF Towel first, and stay away from any cones, surrounds, domes, etc.
Maybe even try something like Simple Green, or another All Purpose Cleaner?
The other is much riskier, and that is to use something like a very small piece of 3M Scotchbrite Pad, and in the same direction as the grain lightly clean, slowly, and again very carefully clean and polish by abrasion.
I would highly caution against using anything like Steel Wool! Not around Magnets! Not sure about sandpapers, there are some that won't easily shed like crocus paper. But others might likely cause their own separate problems, and in any case, you do not want to do more harm than good. Can't stress this enough.
If I were to do such, I would of course remove the drivers. Be careful about wiring of you do.
I know some might say what's important is that all the drivers work correctly, are the correct drivers, etc.
And some might comment and say that when the Grills are on, you don't see that anyway.
I'm not sure if the folks over at the Lansing Heritage Forum might have some better answers than I, so you could pose these questions you have there. Lots of people there with very good experience. Especially with any of the 4300 Series Monitors.