Hi Rolf,
The mechanics of the self tacking jib system is complex and highly variable. With your curved track you get more change in traveller car position as the sail force changes than on a straighter track. Also the mechanical advantage between the jib car and jib clew effects the sensitivity between the jib car movement and sail pressure.
Also with your present system when the jib car moves inboard the jib leech tension is eased at the same time which should tend to let the jib leech open. Like I said, it is complicated.
Bill