Wouter,
You ask some interesting questions but are your really comparing apples with oranges?
Before I get on with my reply, I would like to state that you (and so many others) know far more about boat theory than I do (or probably ever will)

. So the following is an attempt to explain the scenario you discribed based on my current (limited) understanding and intuition. But this won't keep me from trying!

My argument:
These two boats are similar in weight and momentum but different in length, hull/bow shape, mast height, and sail shape/aspect ratio. M20 obviously is longer, has a taller mast, and a higher aspect raio square top sail. I'm not sure of their bow and hull shapes and volumes, but I suspect that the P16 plows/cuts through the waves whereas the M20 bow rises over them more prominently.
When a bow associated with a given mass hits a wave it is impeded to some degree. This impediment by a wave is associated with some amount of raising of the bow by the wave (more pronounced on planers than cutters but present in both). Fist, for the moment let's assume that the two boats in question raise their bows the same amount. Whatever the amount of bow rise, two ramifications would be: 1) loss of forward momentum due to energy needed to raise the bow (approximately the same on both boats due to similarity of mass), and 2) bow rise would be tranmistted to the rig - bow goes up, top of mast rocks BACKWARDS, and this backwards motion of the mast would be MAGNIFIED (proportionally) by a longer hull length (bow to mast base) and mast height. So even if we assume the bow rise of the two boats was identical then the backwards rocking of the M20 mast would be greater. Result: rising bow rocks mast backwards which stalls or detatches sail flow making it impossible to keep the boat powered up or trim the sail during this episode, and this effect is more pronounced on longer boats with taller rigs. Finally, since the M20 has a squarehead sail vs the P16's pinhead, even if the mast lengths were similar, the squareheaded M20 would be further disadvantaged because more of its sail area (proportionately) would be impacted by this effect. Since this effect keeps happing wave after wave, with each wave there is even less power in the sail than the wave before providing less power to drive through the next wave. This allows successive waves to raise the bows even more than they did initially when the boat was more powered up creating a vicious circle. So even a slight increase in masthead rocking in the M20 vs P16 would accumulate quickly due to rapid recurrence making a BIG diffference between the two boats ability to perform in these conditions.
I'm not familiar with the P16 vs M20 hull/bow shapes/volumes but I suspect that the M20 bows are more bouyant due to the greater size of that boat and for all I know it's more of a planer and the P16 more of a cutter. If these are true, than the M20 bows would actually rise slightly more than the P16 despite similar mass and the effects on backward rocking of the mast would be proprotionately magnified even further.
One more issue which would be interesting to understand in this regard is: I would imagine that higher aspect ratio sails (ie, shorter chord) could be more susceptible to stalling or detatched flow than lower aspect ratio sails under similar conditions. Just an instinct. If so, then this would be another factor influencing the different outcome. Anyone know the hydrodynamics of this and if higher aspect sails are more "critical" in this regard?
Jerry