You're putting too much effort into it and the solution is getting lost in the analysis. The system we are evaluating is a static system where the sailor's position (weight) and the boat (including mast) are not moving in relation to each other and can be looked at as one fixed system at any instant in time. While this relationship does change as the boat is righting, we're only looking at the system in a fixed point in time so the forces at play will start to right the boat.

With a static system, it simply breaks down to where the center of mass of the combined system is in relation to the center of buoyancy (rotation). The sailor isn't moving in relation to the boat in this analysis so there's no point in looking at the individual loads on the line related to it's angle. If the line angle does get lower, the line carries more load to support the same weight of the sailor (because the angle is sharper) but that is offset by the beam it's connected to and the amount of force the sailor is having to use to hang onto the line - it cancels itself out of the equation. The line isn't doing anything but supporting the sailor and keeping the system static....so it doesn't matter if the line is higher or lower when you look at the system as a whole. Only if you were trying to minimize the amount of effort you needed to hold yourself up or minimize the amount of loading a fixture on the boat has to hold does the angle of that line come into play. On the system as a whole, the relationship is fixed, non-moving, and the details about the line do not matter.

If having the line over the hull vs. having it attached lower to the beam really did make a difference, all keel boat's keels would look like upside down "A"s since they would also want the support to the keel from the outer most point of the hull....but they don't because it doesn't matter. Only the amount of moment the weight applies to the fixed system comes into play - not how it's supported in the fixed system. Hence, typical keel boats use a slender fin to hold the bulb as far away from the center of buoyancy/rotation to get the most righting moment.

In the diagram below, all that matters to get the boat righted is that Fa+b is on the sailor side of the center of buoyancy marked with a circle and cross. Once it passes that center point, the boat is righting. Note, however, that my formula below is wrong...Fa+b in the formula should read Ma+b to represent the righting moment...weight times distance from the center of rotation. Or it should read Fa+b(Dc) where Dc is the distance the combined sailor and boat weight are from the center of buoyancy.