It depends on what you want your HT to able to do. DTS in many movies actually recommends you use large towers at all four corners for proper sound. Dolby originally, if I remember correctly, assumed the use of small satellites because they were intended to be producing very small amounts of ambient information in a severely restricted frequency range so small was all it needed.
Along with that the Dolby receivers of the day had only a small amount of amp power for the rear channels. However as surround sound took off, this changed a LOT. With DTS emerging to challenge Dolby, Dolby responding with Dolby Digital, etc. things changed pretty quick. Now HT receivers needed full power in all channels to handle the requirements of all the formats and possible speaker combinations. So what started out only needing small satellites to handle the limited sound and frequency range now needed more power and larger speakers.
A good bookshelf speaker will potentially give you better sound if it has good wide dispersion characteristics. That's what helps sound that travels from the rear to the front or from front to back appear more seamless.
Dedicated satellites for rear HT applications can often have multiple speakers on 2 or three sides to improve this affect. Smaller satellites that don't have this, aren't from the same series, etc. can often make this worse than using a decent bookshelf would be able to do.
It all depends on how well you want it all to blend together. And it all depends on how willing you are to take the time to properly set it all up. Positioning is often crucial to hearing a really good movie sound it's best. Along with making sure all the settings in the HT receiver itself is right.
As far as a sub goes, it really depends on what you want it to do. There seems to always be a lot of cheap subs showing up on CL but if they will work in your system, match your speakers sound, much less actually dig down for tight, accurate bass in your listening room can be a crap shoot.
Do the research on line and at least get an idea of what you are looking for with your speakers. Double check the specs for your speakers for how low they can cleanly go and start there. Then look at different deals on line and research each model for reviews, forum threads either here or elsewhere, and check out it's specs.
The best you can do is spend some time digging in to your research. Educate yourself because that's the only way you'll know for sure you're hopefully getting what will work for your system in your room. DON'T be in a rush. Take your time and look around. It will pay off in the long run.
