Loudnoises adds details I omited - use sandpaper, emery board (which is what I use), or a file, and keep the tool vertical so the edge remains vertical. If you gently chuck the outside of the idler's oilite bushing in the drill, you won't damage it, or equally, if you lock the idler bushing on a bolt of the same size as the hole and lock it with a nut, you won't damage it.
But with most idlers, doing it in place is easier and fast enough.