Maybe check a few things:
That you have the pitch range the lowest setting at that moment you can get away with and still reach the needed pitch. You might want to set the pitch buttons to range instead of bend in the preferences/utility to make switching range easier.
Try to pay more attention to the tempo/pitch readouts on screen to notice if this drifting is coinciding with the furthest-right digits fluctuating when you’re not touching the pitch fader.
What you’re experiencing might be related to this thread: