I wear the harness on the outside for the opposite reason. If I get hung up on the hook, I want to be able to get out of the harness quickly.
I wonder if this is why they have the quick disconnect hooks now. I personally put the life jacket on after everything else except the gloves and waterproof ipod.