How to align photos


#1

I try to align red and nir mosaic. But I can’t. Photos have different pixel number after generate mosaic. Also photos are misalignment. How can I solve this problem ?


#2

Hi @blackman

So your question relates to processing outputs of your processing?
I think it would be more efficient to ask your software provider as the answer will be more on point.

If you want to align pictures before mosaicing , I suggest you read this application note


#3

@blackman, red and nir can’t be aligned by normal image sticking methods as they are multimodal images. You will need to align them by relative orientation of two respective cameras.

I would suggest that you study Pinhole camera model and Epipolar geometry first to understand the concept of camera rig relatives which describe the orientation of two cameras in space relative to each other.

Once you know the basic equations and model, you will be able to under the document pointed to in the above comment.


#4

I’m implementing image registration in Python, referencing application note 3.
But my results may be wrong.
Does anyone improve my code?

NIR:

RED:

My source code:

import cv2
import exiftool
import numpy as np

DEBUG = True

rig_relatives_tag = "XMP:RigRelatives"

# T_rel
NIR_T_rel = [15, -15, 0]
RED_T_rel = [0,  -15, 0]

def image_registration(img_path, type):
    # Master Camera parameter
    T_m = [0, 0, 0]
    R_m = rotM([0, 0, 0])

    forcal_length = 4

    img = cv2.imread(img_path, -1)

    height, width = img.shape[:2]

    with exiftool.ExifTool() as et:
        rig_relatives = et.get_tag(rig_relatives_tag, img_path)

    rig_relatives_list = rig_relatives.split(",")
    rig_relatives_array = np.array(rig_relatives_list, dtype=np.float32)

    # Calculate Rotation Matrix
    R_rel = rotM(rig_relatives_array)

    if type == "NIR":
        T_rel = NIR_T_rel
    elif type == "RED":
        T_rel = RED_T_rel
    else:
        print("Error: Image type error, NIR or RED")

    if DEBUG:
        print("Rig Relatives:", rig_relatives_array)
        print("R_rel:\n", R_rel)
        print("T_rel: ", T_rel)
        print("det(R_rel)", np.linalg.det(R_rel))

    # matrix for remap
    map_x = np.empty((height, width), dtype=np.float32)
    map_y = np.empty((height, width), dtype=np.float32)

    for y in range(height):
        for x in range(width):
            # camera coodinate to world coodinate
            u, v = inv_pp(x, y, width, height, forcal_length)

            X = np.array([u, v, 1])

            X_d = R_rel.T.dot(R_m.T.dot(X - T_m) - T_rel)

            # world coodinate to camera coodinate
            x_d, y_d = perspective_projection(X_d[0], X_d[1], width, height, forcal_length)

            map_x[y, x] = x_d
            map_y[y, x] = y_d

    regi_img = cv2.remap(img, map_x, map_y, cv2.INTER_LINEAR)

    return regi_img

def perspective_projection(u, v, width, height, forcal_length):
    f_x = forcal_length / width
    f_y = forcal_length / height

    delta_x = width / 2
    delta_y = height / 2

    x = f_x * u + delta_x
    y = f_y * v + delta_y

    return x, y

# inverse perspective projection
def inv_pp(x, y, width, height, forcal_length):
    f_x = forcal_length / width
    f_y = forcal_length / height

    delta_x = width / 2
    delta_y = height / 2

    u = (x - delta_x) / f_x
    v = (y - delta_y) / f_y

    return u, v

# calculate Rotation Matrix
def rotM(p):
    px = p[0]
    py = p[1]
    pz = p[2]

    Rx = np.array([[1, 0, 0],
                   [0, np.cos(px), -np.sin(px)],
                   [0, np.sin(px), np.cos(px)]])

    Ry = np.array([[np.cos(py), 0, np.sin(py)],
                   [0, 1, 0],
                   [-np.sin(py), 0, np.cos(py)]])

    Rz = np.array([[np.cos(pz), -np.sin(pz), 0],
                   [np.sin(pz), np.cos(pz), 0],
                   [0, 0, 1]])

    R = Rx.dot(Ry).dot(Rz)

    return R

if __name__ == '__main__':
    img_nir_path = "src/IMG_170704_002438_0036_NIR.TIF"
    img_red_path = "src/IMG_170704_002438_0036_RED.TIF"

    img_regi_nir = image_registration(img_nir_path, "NIR")
    img_regi_red = image_registration(img_red_path, "RED")

    cv2.imwrite("img_regi_nir.png", img_regi_nir)
    cv2.imwrite("img_regi_red.png", img_regi_red)

Source Images:
NIR:https://drive.google.com/open?id=1sqTtfflYRbFLLW1HvTocJYahCzevoup9
RED:https://drive.google.com/open?id=1F05FWhMKT9OFBAdigRbHLiyRf-0LVwLw

Thanks.


#5

This is a simple flowchart for image-registration:

Does anyone know where a mistake of the flowchart is?


#6

Hi @muratak,
Are you sure you are using the focal length in pixels?

Also, the transformation you use in betwen image and 3D projection looks different from the one in:


#7

@domenzain thank you for reply.

I don’t know how to evaluate the focal length in pixels well.
Could you tell me it?

I found this page: http://answers.opencv.org/question/17076/conversion-focal-distance-from-mm-to-pixels/
Is this way correct?

By the way, I also found error in my python script.
I didn’t use radian instead of degree when I evaluate sin and cos.


#8

Hi @muratak,

The conversion looks fine.

As @muzammil360 mentions above to the original poster, you should read up on the base concepts to understand the language and conventions of the Application Notes and other supporting documents. They are complete but are not intended as introductory materials.

For most applications Parrot recommends using a photogrammetry solution like Pix4D. Only people with very specific applications should prefer implementing everything themselves as it is highly non-trivial and error-prone.