... should have rights only to a point.
I think the term "rights" is unfortunate. The phrase "right of way" is also misleading. We see those words and think of the rules as entitlements. Then we get upset when people violate our "rights".
If you turn that around and read the rules as obligations, then things get simpler and less heated. A port-tack boat is obligated to keep clear of a starboard-tack boat. A windward boat is obligated to keep clear of a leeward boat on the same tack. A boat clear astern is obligated to keep clear of a boat clear ahead on the same tack. A boat that is changing course is obligated to give a keep-clear boat room to keep clear. All boats are obligated to avoid contact if reasonably possible, and so on.
A boat may be "right-of-way" and still have several other obligations.
I hope that helps,
Eric