Particularly if you have a sealed box, you need some fiberglass, polyester or foam in there, both to minimize resonant modes in the box and to make the box act like a larger enclosure. If you take out the stuffing, you will probably have a louder but more colored speaker. You may or may not like this. It also may affect power handling. In many cases, the designers actually knew what they were doing, and second guessing them may not be such a good idea. But its one way of learning how speaker design works. But let me suggest two things: keep the stuffing you removed in a marked bag, so you you replace it if you change your mind, and make sure you use caulk or foam to keep the driver seal air tight -- if you create leaks, you are moving into a completely different speaker topology, and it will probably be inadvertant, so you won't know why the sound has changed.