I would use a FEM code (http://en.wikipedia.org/wiki/Finite_element_method_in_structural_mechanics).
For a simple rule of thumb method, I have to know the geometry of the rig. For a beach cat, you could assume to have the weight of crew, helmsman and half boat weight on the shroud, apply a margin of safety of 3...4, then you have a rough idea of the loading. E.g. 2-up beach cat of 100kg weight: (2 x 80 + 50) x 3 = 630 kg. This requires at least a 3mm wire. Keep in mind that a margin of 3 is low, and requires a regular check. Lazy guys could go with a 4mm wire and less regular checking.

Cheers,

Klaus