I think it should be as simple as making factual information available about the judges. There are many more times over the number of unreasonable sailors as there are judges who have poor judgement. I wouldn't trust a sailor review process very much. Stick to the numbers. How experienced is this judge?...i.e. How long have they been accredited, how many regattas have they be involved with (it would be nice to be able to gauge the size/significance of the regattas somehow), how many cases have they heard, and specifically - how many rulings have been overturned by a higher process?
My vision would be the customer (the sailors) would file the complaint, then a qualified board (peer review board) would review and rule on the complaint, this really applies to Race Officers. For judges this process is already in place (appeal). What is not available as you pointed out is how often an appeal has been filed against a judge's ruling and how many times that judge has been overturned.
I also agree it should be as factual as humanly possible.
