You shouldn't sand down a pulley. That will shave metal and reduce the speed. Instead, you should use isopropyl alcohol on the pulley with a microfibre cloth.
This stuff is comparable to like 1500 grit (maybe a little finer than that) but I will keep that in mind for future reference.
You know, I'm not so sure Maraakate is off-base here or even treading on iffy waters.
I've often wondered if a little bit of sanding down might be good for the pulley on older idler and belt drive motors. On the four Empire's that have passed through my hands over the years I noticed all of them had pulleys that were on close inspection "encrusted" with rubber and other airborne pollutants and debris compacted on them. Alcohol alone never got them clean; it took a little emery cloth and some noxon/brasso to get them smooth and shiny again. The belt really seemed to grip the pulley afterwards again too, as a result. Plus, those pulleys seem to be really hard. I think it might be very difficult to wear them down enough to alter speed that radically.
In fact, I'm starting to wonder if instead of us trying to find belts that work perfectly with our Empire's, we should go the other way: find a good quality belt that lasts, fits, is fairly priced, widely and consistently available, and consistent in the speed error it produces on the tables, then alter our pulleys to make it work perfectly. A little "don't raise the bridge, lower the river." Hmmm.....