I have had much the same problem as you with one of my bows. The other bow tuned fast.
A couple of things I have found is that moving the rest up and down affects your right and left some also. It can be a tricky combonation. Another thing you might try is; if you are using one broadhead arrow, set up another one and shoot it too. I have found that I need to use multiple tuning arrows because some just don't fly true, and usually it is the nock that is to blame.
I am not a tuning Guru, I strugle with it some times just like you. Only trying to help.