I forgot to say that I finally figured this problem out.
For ease of use, I had set one of the buttons on the pen to the hand tool, so I could easily move the image around when zoomed in.
That was apparently the issue. I changed both pen buttons to be off and now the problem is no more.
Over ten years of trouble due to that little stupid thing.

And that's also why the problem only happened in Photoshop.
It has something to do with how PS handles that issue.
Anyways, that's solved. Finally.

Maybe I'll post some sketches here this week.