First, the only time the term right of way is used in the Navigation
Rules,
is in regards to a vessel coming down a river.
Pleasure craft don't have to yield to commercial craft.
Everyone has to avoid a collision.

************

That statement is so broad as to be misleading. It's almost as loose as
"sail always has the right of way over power."

Without looking up the exact number of the Col Reg, there's an
important one that says "no vessel under..... (about 65
feet....20meters?) and no vessel under sail......(regardless if it's a
150-footer)....shall impede any vessel following a VTS channel." As any
boat can follow the VTS if it so chooses, this reg "could" be stretched
to say that any vessel under sail must yield to any vessel under power
in the VTS- but it never is. In most situations where it could be an
issue, "pleasure boats" under 65 feet and all boats under sail
regardless of LOA must stay out of the way of commercial vessels.

You did the correct thing by backing off. No point being "dead right".