Craigslist; Thrift Stores-Salvation Army, Goodwill, etc.; Garage Sales, etc.
It all depends what you want to do. You can probably put together a half decent system for no more than $200.
Right now I'm listening to Wes Montogmery Jazzmasters Verve Compilation on a SONY DVP-S7700 TOL DVD player ($20) from the year 2000 going into an Onkyo 303 AV receiver ($90 refurb). The speakers are 2 Boston Acoustic Bravos ($50) and Pioneer SP-BS22-LR ($80) with a Yamaha YST-215 Sub for Bottom End ($30?-It was a package deal). Let's see that is about $270 but that is because some of those items were bought new albeit at discount prices. I have a system in the Dining Room (Infinity Kappa 5 speakers, Sony DVP-NS90V DVD/SACD player and Luxman R-113 Receiver which I believe was $90 total. I also have some items that I could cobble together to create a full system with turntable, CD player, Receiver, and reasonable speakers that cost altogether less than $10.
The real trick is to not get caught up in what is 'high end'. For example I don't buy expensive interconnects or speaker wire. I don't think it makes a huge difference in MY musical enjoyment.
That is the essential question. You may or may not need an expensive high end system to enjoy your music, but you could certainly start with something cheaper and work your way up.