Wondering if you might get a better smooth finish if you just started by using a reduced poly finish instead of glue to set the paper? Saturate the paper completely, apply, roll flat, repeat as needed, then once you get into sanding, you've got a consistent hard surface to work with. Wouldn't expect anything less than a mirror smooth finish doing it that way.
And yes ... patience IS a good thing if you want an OCD friendly finish. I worked this case to 1200 grit - I can see myself!