FIRST, THE SELDOM-THOUGHT-OF STUFF:
No one has mentioned the ABSOLUTE MOST IMPORTANT element of all...
That pinkish-greyish, wrinkly pudding-esque thing called a
brain that sits between, and is connected to, your ears. Without it, NOTHING is heard, sensed, felt or even imagined, let alone enjoyed. You can spend your listening time so over-analyzing the sonic characteristics of the system, or so preoccupied with tomorrow's presentation at work, or perhaps sound asleep or even unconscious in a coma, that the whole experience is wasted.
Another important thing, also un-mentioned, is the air that carries the sound vibrations from the transducers (drivers in your speakers or headphones) to your ears. [EXCEPTION: The Bone Phone is being ignored here...

] The old line is true:
In space (i.e., in an airless vacuum),
no one can hear you scream... let alone a Dvorak symphony or a Beatles tune! Music sounds different when transmitted through pure helium or nitrogen or any other pure or mixed gas, other than Earth's air, and may even sound different when the air is at different pressures and humidity levels. We tend to ignore this, because we can do precious little about it, in practical terms.
Related to this, is the room in which the air is confined. Sound waves passing through the air will reflect off of the various surfaces they encounter, and this creates interference patterns, peaks and nulls that affect what we hear. Depending on what these surfaces consist of, and where they are located vis-a-vis the speakers and our ears, the effects may differ greatly, giving us everything from "extremely lively" rooms to dull rooms to anechoic chambers. Only if you could suspend both you and your system well above the Earth, hanging in open space, could you eliminate the effects of the room. Anechoic chambers are an attempt to simulate this condition. The best-sounding system will probably exist somewhere between the extremes, with a "pleasing" balance of reflections that help our brain interpret sonic clues.
The fourth thing that is VERY IMPORTANT is probably the incoming electricity itself. Feed a really noisy, varying voltage --perhaps current off a circuit with multiple fan speed controllers, air conditioners, etc... off a shaky, third-world-quality grid-- into into an otherwise great audiophile system, and you won't be very pleased with the result (provided adequate power conditioning isn't built into the gear). Feed a perfect signal into an average system, and you'll be surprised at how much the sound can improve.
NOW FOR THE USUAL SUSPECTS (and a few other thoughts...):
As for the system itself, actually, every element is critical. Just try removing any one part, even the thinnest power cord or interconnect or speaker connector, or just one tiny fuse, and wait for the electrons to jump through the air on their own, instead. Just see what happens to your sound --it disappears!
If you are asking "where should I start to upgrade, to provide the greatest improvement in my system's sound?", the two MOST COMMON areas that need to be made as good as you can get them, are the two
transducers at either end of the system that Tom Brennan mentioned: Your TT's cartridge/tonearm system (and to a fair degree the TT, including plinth/base they are attached to, since all of these interact as a system) at the source end, and the speakers at the other end.
Significant upgrades to either of these will almost certainly make the biggest, most obvious improvement in your system's sound. Why? Because these are almost always the "weakest link", and to make them truly perform well is a challenging and expensive design and engineering challenge. These items usually add the most "coloration" and "distortion" to the music.
Which is more important -the source or the speakers? This is a complex issue which --despite many claims to the contrary-- probably has no absolute and final answer. For YOUR system, in YOUR environment, with YOUR music, ask yourself which one is the weakest link?
Personally, I started with great speakers, and only much later got a truly good source (still subject to further upgrades!), and I'm glad I did it that way, but over time it probably would have worked just as well the other way around!
Why? Because without very good speakers, you cannot really hear what is being fed to them, anyway. A spectacular source signal still sounds mediocre through mediocre speakers. Yet without a good source, your speakers can only reveal the limitations of the source, and let you hear the flaws, and not show you what isn't there in the first place. You need BOTH to be good -with nothing detrimental in between-- to achieve anything close to real "sonic bliss".
One of the ironies of this whole subject is that the closer your system gets to being truly high-end, very revealing, transparent and "audiophile", the MORE EACH LITTLE THING MATTERS. In a truly outstanding, near state-of-the-art system, the effect of replacing one single jumper wire on the back of a speaker can make a significant, audible difference. [I've heard this first-hand, and posted about it here.]
But that same change would be lost, inaudible, in a system that wasn't already very good, and consistently so throughout most of the audio chain. One truly bad "weak link" that distorts, adds noise, causes information to drop out, etc... will limit the performance of the entire system. You cannot replace sonic information, once it has been lost, at any step in the chain.
Another consideration emerges here, that of "synergy" between components. What is it, really? In most cases, it means using the colorations and distortions of one component to offset the opposing colorations and distortions of another component! If your digital source sounds too "shrill", use a speaker with rolled-off highs to eliminate the harshness on top, for example. The problem with this is that you simultaneously eliminate some of the high-end information, and with that some of the sound stage and realism disappears, gone forever. [Another aspect of synergy that is sometimes ignored to some detriment is matching impedances within appropriate/compatible ranges. This issue is especially important at the source end of an analogue system, especially if you use LOMC cartridges. Most other components (eg, preamps and power amps) are close enough for this not to be much of an issue.]
What if you then take the opposite tack, and try to make your system as revealing and accurate as possible? Then, on the best, world-class recordings, you will hear sublime things... and on every other recording you own, you will hear all the sonic flaws and warts and poor mixing and mastering decisions, etc... and won't be able to enjoy the music as well. My personal answer to this is to have one revealing system for the best recordings, and a second, more forgiving one, for poorer-quality source recordings. Another approach might be to make an ultra-revealing system first, then selectively introduce elements (perhaps a tube buffer here, an EQ adjustment there) that make it more forgiving and more "musical", up the point where the trade-offs made suit your personal tastes and listening habits/preferences.
Along the path to seeking sonic perfection, improvements that are significant may actually seem minor, because some other inferior component elsewhere in the chain prevents you from hearing their real impact. This is a second reason why it is so important to have truly good transducers at the ends: you cannot hear the effects of anything if you don't have a good signal to begin with, or if your speakers aren't letting you hear everthing that is (or isn't) there.
So keep upgrading where and when you can, keep on tweaking and keep on testing (for as long as your interest and patience hold out) and keep on upgrading. You may sometimes find that one component seems to make a "great leap forward" in your system's performance, but perhaps it is just eliminating a bottleneck point and allowing the full extent of other improvements to be heard.
In a similar vein, as the saying goes "God lies in the details." Every little improvement you make has a contributory effect, and sometimes many small improvements can cumulatively add up to a very significant improvement. For example, if you haven't done much tweaking to your system (and MOST people haven't!), go through your system and DeOxit all contacts, including the power-socket connections and fuse connectors, etc... Arrange all your power cables, interconnects, and speaker wires to keep them spaced apart as much as possible, and crossing at 90 degrees (and separated by some distance) if they absolutely must cross. Add isolation (or coupling, as appropriate) under each component, and add mass/weight and damping where appropriate. Add shielding where appropriate. etc... etc... Then listen again to your system. Similarly, make sure your speaker placement is optimal, and try "treating" your room (with things like bass traps). Listen again. None of these things need be expensive, or even very difficult to do.
The cumulative effect of such small things can be massive and dramatic. Why? Much of the "magic" that we hear in really good systems comes from our ability to hear tiny, clear, discrete detailed bits of sonic information. Unblurred, unsmeared, un-interfered-with, clearly-reproduced information. Every single thing we do that reduces vibration, clears out the roadblocks of oxidative resistance, reduces interference (whether from outside our gear or between parts of it), and allows a cleaner transmission of the original signal through the system and up to our ears, helps that detail to fully come through so our brains can interpret and decode it back into an impression of the original live performance.
Along these lines, if you are using vintage gear with 20 or 30-year old electrolytic capacitors and other poorly-aging components (like carbon resistors that can grow "noisy" as they age), that is probably the single biggest "bottleneck" in your entire system. Each of those poor components subtracts a little bit from the sonic potential of your system's performance. It may be a much wiser expenditure to get your gear properly overhauled by a really good tech, than to buy another "upgrade".
This whole area of the hobby is a controversial subject, often because people lose sight of the big picture, while focusing on the small, single changes. Try to maintain the overall perspective and not be fooled by what you THINK you are hearing in the first instance. We are all influenced by our own experiences. If you replaced your speakers and the whole system was transformed, you'll believe that speakers are most important. If you replace your turntable and your system is likewise transformed, you'll believe that the source is most important. But perhaps in the first instance, the speakers were the weakest link, and the source wasn't so bad, or in the second one, the source had been the weakest link. Or perhaps you are inadvertently achieving some kind of better synergy with the substitution, and that is the main reason for the big improvement.
It is this very complexity, and the nearly-infinite number of possible variations in set-ups/systems/approaches that are possible, that keeps this hobby so fascinating and rewarding. Fortunately, there are many roads to the end of the rainbow, and many different pots of gold for us all to savor and enjoy along the way.
I've still left out a few major thoughts and considerations related to this topic, but this has already become another overly-long post, so I'll stop here.