I try to align red and nir mosaic. But I can’t. Photos have different pixel number after generate mosaic. Also photos are misalignment. How can I solve this problem ?

Hi @blackman

So your question relates to processing outputs of your processing?

I think it would be more efficient to ask your software provider as the answer will be more on point.

If you want to align pictures before mosaicing , I suggest you read this application note

@blackman, red and nir can’t be aligned by normal image sticking methods as they are **multimodal** images. You will need to align them by **relative orientation** of two respective cameras.

I would suggest that you study **Pinhole camera model** and **Epipolar geometry** first to understand the concept of **camera rig relatives** which describe the *orientation of two cameras* in space relative to each other.

Once you know the basic equations and model, you will be able to under the document pointed to in the above comment.

I’m implementing image registration in Python, referencing application note 3.

But my results may be wrong.

Does anyone improve my code?

NIR:

RED:

My source code:

```
import cv2
import exiftool
import numpy as np
DEBUG = True
rig_relatives_tag = "XMP:RigRelatives"
# T_rel
NIR_T_rel = [15, -15, 0]
RED_T_rel = [0, -15, 0]
def image_registration(img_path, type):
# Master Camera parameter
T_m = [0, 0, 0]
R_m = rotM([0, 0, 0])
forcal_length = 4
img = cv2.imread(img_path, -1)
height, width = img.shape[:2]
with exiftool.ExifTool() as et:
rig_relatives = et.get_tag(rig_relatives_tag, img_path)
rig_relatives_list = rig_relatives.split(",")
rig_relatives_array = np.array(rig_relatives_list, dtype=np.float32)
# Calculate Rotation Matrix
R_rel = rotM(rig_relatives_array)
if type == "NIR":
T_rel = NIR_T_rel
elif type == "RED":
T_rel = RED_T_rel
else:
print("Error: Image type error, NIR or RED")
if DEBUG:
print("Rig Relatives:", rig_relatives_array)
print("R_rel:\n", R_rel)
print("T_rel: ", T_rel)
print("det(R_rel)", np.linalg.det(R_rel))
# matrix for remap
map_x = np.empty((height, width), dtype=np.float32)
map_y = np.empty((height, width), dtype=np.float32)
for y in range(height):
for x in range(width):
# camera coodinate to world coodinate
u, v = inv_pp(x, y, width, height, forcal_length)
X = np.array([u, v, 1])
X_d = R_rel.T.dot(R_m.T.dot(X - T_m) - T_rel)
# world coodinate to camera coodinate
x_d, y_d = perspective_projection(X_d[0], X_d[1], width, height, forcal_length)
map_x[y, x] = x_d
map_y[y, x] = y_d
regi_img = cv2.remap(img, map_x, map_y, cv2.INTER_LINEAR)
return regi_img
def perspective_projection(u, v, width, height, forcal_length):
f_x = forcal_length / width
f_y = forcal_length / height
delta_x = width / 2
delta_y = height / 2
x = f_x * u + delta_x
y = f_y * v + delta_y
return x, y
# inverse perspective projection
def inv_pp(x, y, width, height, forcal_length):
f_x = forcal_length / width
f_y = forcal_length / height
delta_x = width / 2
delta_y = height / 2
u = (x - delta_x) / f_x
v = (y - delta_y) / f_y
return u, v
# calculate Rotation Matrix
def rotM(p):
px = p[0]
py = p[1]
pz = p[2]
Rx = np.array([[1, 0, 0],
[0, np.cos(px), -np.sin(px)],
[0, np.sin(px), np.cos(px)]])
Ry = np.array([[np.cos(py), 0, np.sin(py)],
[0, 1, 0],
[-np.sin(py), 0, np.cos(py)]])
Rz = np.array([[np.cos(pz), -np.sin(pz), 0],
[np.sin(pz), np.cos(pz), 0],
[0, 0, 1]])
R = Rx.dot(Ry).dot(Rz)
return R
if __name__ == '__main__':
img_nir_path = "src/IMG_170704_002438_0036_NIR.TIF"
img_red_path = "src/IMG_170704_002438_0036_RED.TIF"
img_regi_nir = image_registration(img_nir_path, "NIR")
img_regi_red = image_registration(img_red_path, "RED")
cv2.imwrite("img_regi_nir.png", img_regi_nir)
cv2.imwrite("img_regi_red.png", img_regi_red)
```

Source Images:

NIR:https://drive.google.com/open?id=1sqTtfflYRbFLLW1HvTocJYahCzevoup9

RED:https://drive.google.com/open?id=1F05FWhMKT9OFBAdigRbHLiyRf-0LVwLw

Thanks.

This is a simple flowchart for image-registration:

Does anyone know where a mistake of the flowchart is?

Hi @muratak,

Are you sure you are using the focal length in pixels?

Also, the transformation you use in betwen image and 3D projection looks different from the one in:

@domenzain thank you for reply.

I don’t know how to evaluate the focal length in pixels well.

Could you tell me it?

I found this page: http://answers.opencv.org/question/17076/conversion-focal-distance-from-mm-to-pixels/

Is this way correct?

By the way, I also found error in my python script.

I didn’t use radian instead of degree when I evaluate `sin`

and `cos`

.

Hi @muratak,

The conversion looks fine.

As @muzammil360 mentions above to the original poster, you should read up on the base concepts to understand the language and conventions of the Application Notes and other supporting documents. They are complete but are not intended as introductory materials.

For most applications Parrot recommends using a photogrammetry solution like Pix4D. Only people with very specific applications should prefer implementing everything themselves as it is highly non-trivial and error-prone.

Hi@muratak

I have used opencv for python alignment, but the result is not good enough.

How about your python script for sequoia’s image registration. Is it works fine right now?

If it does. Could you share your python script for image registration?

Colmap https://github.com/colmap/

+

Aerial Mapper https://github.com/ethz-asl/aerial_mapper

is like Pix4fields for image registration

Pix4fields is mostly based on Eigen AKAZE feature tracking algorithm ( http://metrology.survice.com/sites/metrology.testing.survice.com/files/cmsc-16-initial.pdf ) and Ceres Solver (Dogleg trust region methods and SPARSE_NORMAL_CHOLESKY with the Tukey biweight loss function which aggressively attempts to suppress large errors.).