Yes, you are correct. I assume by now you would have noticed that the trick is backpropagation. You start with undistorted points in master plane and add anomaly (i.e. distortion/mis-registration) effect using your mathematical model. Then you pick up pixel values at those distorted pixel locations and just put them in actual place you started with.
And since it’s not necessary that the distorted pixel locations are integer, you might wanna use bi-linear /bicubic interpolation method (OpenCV has support for many).
I had written some stuff based on that to make it easy for people to understand. If I found some time, I would try to extract that and post it here for other people.