I don't know, wouldn't winglets increase the drag to leeward? And all the planes you see them on operate at fairly constant angles of attack, with the way cat masts move around it seems like it would add more drag, for example every time the bows go up and down over a wave it would raise and lower the angle, and keep shaking the airflow off. As a longtime hangglider pilot I've seen different tries at the winglet idea, and nobody has been able to make it work, mostly because every time the wing yaws a little it brings one side to too low an angle as the other side goes to too high an angle. One of those ideas that seem good but that don't work in actual use, maybe?
Maybe, maybe not.
Handgliders are closer to boat speeds, but don't have the same problem of flow attching/detaching near the top of the sail (wing tip) due to waves. It could just happen that a winglet would guide the vortex and help the flow remain attached all the way to the top when the sail moves due to waves. I really don't know and my guess is that even an experienced aerodynamics specialist will ask for wind tunnel tests before giving a final answer.
