I'm not sure that "yes" is the definitive answer, I think it really depends on what speaker and what amp.
For example; a "40 watt" Sunn 2 X EL34 amp is putting out more than 40 watts wide open. On the other hand, amps are generally only outputting max power in spurts rather than continuous power, and also I've seen speakers handle considerably more than the rated power if it's getting a clean signal.
Another factor is that while generally you can hear when a speaker is struggling with too much power mechanically, if it's getting overloaded thermally all you will hear is nothing when finally it melts down. A higher wattage, lower sensitivity driver is generally more likely than a higher sensitivity lower power driver to melt before it farts.
I think it's a balance of the two, but that the sensitivity of a speaker is more critical to "loud" than the ultimate power rating. Contrary to popular belief, a higher power rating does not mean the speaker is louder, in fact it is more often the opposite.
