If you move the mainsheeting point foward on the sail, you may flatten the bottom but you will also induce hell of a lot more twist. the large amount of twist that you say you already have will be a good reason that the boat is not pointing [Pointing is down to jib and main leech control/tension]
It looks like you need to let the mast bend more to take some of the fullness out. i think that the 5.8s go with a 'soft' rig so you can loosen the diamond wires a bit. To stop this bend choking the slot, let some rotation off.
The trouble is, you may end up with a totally gutless boat with no power on this set-up!

New sails time maybe