Check your rudder alignment or for any bends in the mast.

Also realize that the wind shifts faster than the waves so you see different wave conditions on each tack. If the wind shifts right (veer) you will hit waves more head on on port tack than you will on starboard. If you are hull flying, then you are likely seeing significant waves.