One thing I noticed trying to swim to the boat with all that crap you're wearing, really slows you down. I had a vhf, epirb and all the other required stuff. I still would have headed to the beach before using any of it.
I agree 100% on both points. Even with minimal sailing apparel and a PFD, don't expect to swim any faster than your average puppy. I consider myself a relatively strong swimmer, but I can barely keep up with a 2-knot drift with my gear on.
And an old habit from fire-rescue days is that I would be so embarrassed to declare an emergency (and have my staff come pick my keister up) that it had to be something akin to an amputation, open head injury, sucking chest wound or the like to pick up the phone...
So if shore is in sight and I'm not currently being eaten by a shark, I'd probably save the VHF or cell phone until I was standing on the beach (other than to perhaps call Race Control and let them know I'm okay and will coordinate retrieval later on)
And if I was being eaten by a shark, my stuff floats so someone could probably get a free GPS, VHF and peanut butter sandwich...