You are probably fine for a good while as the variations of brass (there are MANY) and the stainless steels are relatively close together on a galvanic corrosion scale. However, I would still unthread the brass toggle from time to time to inspect the threads - especially if you sail frequently in the ocean because corrosion will take place where the stainless and brass are in contact. For reference, stainless and aluminum are practically on opposite ends of the scale (the farther apart metals are on this scale, more corrosion will result from their mechanical contact).