The thing is, you need the power for every time you play your system, not just "normal" times. If you
ever crank it up, then you need power for those occasions. Or, to put this another way, it is not the average power that matters; it is the peak power that matters. Whatever is the highest peak you
ever want to listen to is the amount of power you need. Anything less and you cannot play it that loud on that occasion.
So, if once a year you decide to play things very loud, then, for that once a year, you need whatever power it takes for that occasion.
Since it can be difficult to know precisely how much power that is, typically, it will be a good idea to get a little more capability than you ever actually need. Having more power capability than you need does not hurt the sound, but having less than you need does.
Many years ago, I picked up a Pioneer SX-780 receiver that had power meters, and it was instructive, in that, most of the time, with the speakers I was using with it, it would be under a watt. But when I wanted to turn it up, the meters kept me from going as loud as I wanted, because I was reaching the specified limits of it. When the demand for more power occurs, it rises very rapidly due to needing to double the power for each 3dB increase in volume. Most of the time, with those speakers, it was enough power. But it was not enough all of the time. I sold the SX-780 a few years ago.
In my main system, I am presently using different speakers than I used with that receiver, of lower sensitivity, and nominally (and honestly) 3 ohms. Because I previously did not have an amplifier rated for such a low impedance, I bought a power amp to drive them that was rated for such use (and lower). Since I wanted to make sure I had enough power, and because I wanted to not have to upgrade in the future, for any speakers I might reasonably own in the future, I bought the most power I could get at what I considered to be a reasonable price, with certain other requirements (e.g., I did not want anything with fans, because passive cooling is quieter, etc.).
So my current amplifier is probably more powerful than I need it to be. That, however, is not a problem. What is a problem is not having enough power, because then you risk problems due to clipping and you do not get the volume you want.
As for what people need, speaker sensitivities vary so much that the amount of power that different people need can be vastly different from each other, based on that alone. Additionally, the size and acoustics of the room, as well as how far one is from the speakers greatly affects the amount of power needed. And then, of course, different people have different inclinations for how loud they want their music to be, which, again, significantly affects the power that different people will require.
As for what is safe, you might want to get an SPL meter (there are also apps for smart phones, though I have no idea how accurate that would be) and look at various sites, such as:
https://www.osha.gov/laws-regs/regulations/standardnumber/1910/1910.95
One thing to keep in mind with such sites is that they do not tell you what is safe; they tell you what is
KNOWN to cause damage, and the standards for OSHA allow for what they regard to be an acceptable amount of hearing loss. In other words, if you do not want any damage to your hearing, you need to be more careful than those standards. How much more careful is difficult to know. Part of the reason for that is that some of the damage to hearing is typically not immediately apparent, and shows up later in life. In that way, hearing damage is a lot like brain damage from things like boxing and football, in that the full effects are typically only apparent years later.