Clearly, things change over time. Even in the dry, still environment of an Egyptian tomb, change occurs, albeit slowly.
With speakers, wood dries, becomes stiffer and probably more brittle and certainly more resonant. Glues and varnishes dry and perhaps contracts, crack and migrate a little, changing their resonance and the characteristic way that vibrations pass through them. Wires and other metals --baskets, magnets, and more notoriously, the contact surfaces in tone control L-pads and pots-- may oxidize on the outside. Capacitors, we all know (especially electrolytics) change and dry out, virtually always for the worst. Even resistors and other discrete components change, albeit more slowly. Even the paper from which old driver cones were made, dries out.
Each of these changes is bound to have some effect, small or large, on the final sound.
MOST of these changes are negative. Dried-out cases with shrunken glue may become more porous to air, and need re-sealing. Their lesser weight and greater resonance, due to loss of moisture, makes them less resistant to movement. While that is a good thing in a violin, it usually isn't in a speaker -- UNLESS the particular resonance involved is adding a desired "warmth" or "tone" to the speaker. Many speakers are well-liked because of their particular sonic characteristics, which may not be accuracy so much as pleasant-sounding distortions and harmonics. If the effects of drying enhance these pleasant sonics, then the aging process may make the speaker sound better.
However, this is very much likely to be the exception, and NOT the rule.
Similarly, it is possible that a dried-out driver cone may be stiffer, more rigid than one with higher moisture content (think of green, flexible wood versus old, dried boards), which might actually make the cone more accurate. However, it is just as likely that the fibers may be weakened by the effects of age, breakdown from atmospheric pollutants, etc... and the result may be sonic degradation.
Almost everything else I mentioned above, and everything else I can think of, is likely to be a negative factor. We want speaker cases to be heavy and inert. Drying out -- or even the reverse, the "dry rot" of too much moisture in damp climates-- works against that.
Surrounds are notorious for going bad over time: cloth ones need re-sealing, and foam ones crumble. No improvement there.
Oxidized wires suffer from negative "skin effect" and the sound is degraded. This applies both to the internal wires going to/from the crossover and drivers, and to the wires inside the voice coils.
Magnets won't become stronger, unless deliberately subjected to a magnetizing process, something which just won't happen when they are mounted inside of a speaker! On the contrary, a magnet may well very slowly lose it's magnetic strength over time, a natural tendency towards entropy that exists in them. This may or may not be noticeable within a human lifetime, but it may be very noticeable after relatively few years, depending on the type of magnet and the conditions it is stored under.
Most of what we do to "restore" speakers is aimed at bringing them back to a state that is closer to new. We oil the cabinets, put in new capacitors and wires, re-seal or replace the surrounds, clean or replace (upgrade) the connectors. I think this illustrates something basic: for the most part, time is NOT friendly to speaker performance!
Eventually, all of these speakers will simply die. I can't imagine a 1,000 year old paper cone being driven hard, without starting to fall apart. Our hope is that, with a little care and occasional restoration, they can still be kept singing for many years to come, before they finally give up the ghost. One also hopes that the alternatives that will then be available, will be at least as good, sonically. At the high end, I'm optimistic they will be. As long as people have ears and emotions, there will be some demand for good-sounding music, and the engineering knowledge will not entirely disappear. Some specific things, though (Beryllium midrange drivers, for example) may disappear for long periods; such things should be guarded and preserved zealously, as their like may never be seen again in our (or our children's) lifetimes [In this example, JM Labs tweeters notwithstanding.]
All that said, a TOTL vintage speaker, nicely restored, will give much better sound than all but a very expensive (close to equally TOTL) speaker today, so even if its sound may not be quite 100 percent of the sound it had when new, it still represents outstanding value-for-money.
But when you are going back to the 1960s and earlier, many people are taking out the old drivers and putting them into newer, updated cabinets. Some of the old cabinets still sound great, but we understand speaker cabinet design and construction much better now than we did then, and have excellent materials available; the sound that often results from re-housing those drivers is proof of that.