HFOV and VFOV probably not correctly reported in photo mode

[Neil]

I’m streaming video from the drone using the YUV callback in my python app. Up to now I was focused to use an OpenCV calibration file, derived from the drone’s camera for photo and video resolution. Luckily I found this link Getting Front Camera Pose & Intrinsic Parameters, which suggests, that it could be possible to derive camera intrinsics from the vmeta frame delivered with each frame. I have to say, that works great, but — just almost

OK, let’s suppose, I’m using the sample code here for streaming: https://github.com/Parrot-Developers/olympe/blob/master/src/olympe/doc/examples/streaming.py

I have changed the yuv_frame_processing like so, that I write the results obtained from vmeta and info as well as the image itself to disk (this is all more or less your code or code which can be found, so I don’t reveal secrets here):

cv2_cvt_color_flag = {
     olympe.VDEF_I420: cv2.COLOR_YUV2BGR_I420,
     olympe.VDEF_NV12: cv2.COLOR_YUV2BGR_NV12,
}[yuv_frame.format()]

vmeta = yuv_frame.vmeta()[1]
hfov = vmeta['camera']['hfov']
vfov = vmeta['camera']['vfov']
        
info = yuv_frame.info()
height, width = (  # noqa
    info["raw"]["frame"]["info"]["height"],
    info["raw"]["frame"]["info"]["width"],
)
cv2.imwrite(f"test_{width}_{height}_{hfov}_{vfov}.jpg", cv2Frame)

So, what I was doing with this is to start in video mode with a real Anafi 4k at zoom 1. As expected I got a file like this:

test_1280_720_1.2042771577835083_0.7377424836158752.jpg

Definitely 1280 x 720.

Then I changed the zoom from 1 to 3 and the number of images increased on disk, until it finally just left one (terrible) image like this:

test_1280_720_0.45044493675231934_0.2563452422618866.jpg

HFOV and VFOV have changed of course. Literally I can follow the down steps by observing the filenames going down the ladder (upwards sorted here of course)

test_1280_720_0.45044493675231934_0.2563452422618866.jpg
test_1280_720_0.4505131244659424_0.2563452422618866.jpg
test_1280_720_0.45071765780448914_0.2564815878868103.jpg
test_1280_720_0.45099034905433655_0.256617933511734.jpg
test_1280_720_0.45126307010650635_0.25682246685028076.jpg
test_1280_720_0.4517402946949005_0.2570270001888275.jpg
test_1280_720_0.45208117365837097_0.25723153352737427.jpg
test_1280_720_0.4524902403354645_0.25750425457954407.jpg
test_1280_720_0.45324018597602844_0.2579132914543152.jpg
test_1280_720_0.4538537859916687_0.2583223581314087.jpg
test_1280_720_0.4551491439342499_0.25907230377197266.jpg
test_1280_720_0.45630815625190735_0.25975409150123596.jpg
test_1280_720_0.45992153882980347_0.2617993950843811.jpg
test_1280_720_0.46258044242858887_0.26343563199043274.jpg
test_1280_720_0.4661937952041626_0.2654809355735779.jpg
test_1280_720_0.47369328141212463_0.269912451505661.jpg
test_1280_720_0.4846697449684143_0.27632108330726624.jpg
test_1280_720_0.5001459121704102_0.2853885889053345.jpg
test_1280_720_0.5211443901062012_0.29772862792015076.jpg
test_1280_720_0.5387340784072876_0.30809152126312256.jpg
test_1280_720_0.570777177810669_0.3271128833293915.jpg
test_1280_720_0.639431357383728_0.3682235777378082.jpg
test_1280_720_0.7059720754623413_0.40865248441696167.jpg
test_1280_720_0.7786486744880676_0.4535810649394989.jpg
test_1280_720_0.8345537185668945_0.488760381937027.jpg
test_1280_720_0.871641993522644_0.5123496055603027.jpg
test_1280_720_0.9259108304977417_0.547460675239563.jpg
test_1280_720_0.9784752130508423_0.5819582343101501.jpg
test_1280_720_1.028926134109497_0.6156376004219055.jpg
test_1280_720_1.0911716222763062_0.6581118702888489.jpg
test_1280_720_1.1437361240386963_0.6947228312492371.jpg
test_1280_720_1.1736657619476318_0.7158576846122742.jpg
test_1280_720_1.1878465414047241_0.7260160446166992.jpg
test_1280_720_1.2001866102218628_0.7348790764808655.jpg
test_1280_720_1.2040045261383057_0.7376061677932739.jpg
test_1280_720_1.2042089700698853_0.7377424836158752.jpg

So far so good. Now it comes…

I changed to photo mode and the next image I got (and all afterwards) have been tagged to be 1024 x 768 (OK) with the HFOV and VFOV of the last zoom stage 3 of the video (this is, what I would consider to be not OK).

test_1024_768_0.4503767490386963_0.25627705454826355.jpg

While the image is really 1024 x 768, I can’t believe the HFOV and VFOV.

Could you please try to elaborate?

[Neil]

As a little side issue I was noticing different HFOV and VFOV, but just if I use your official app to switch between photo and video.

In video mode the vmeta for HFOV, VFOV reads like so:

(<VMetaFrameType.PROTO: 6>, {‘drone’: {‘quat’: {‘w’: 0.8680419921875, ‘x’: 0.0211181640625, ‘y’: 0.03900146484375, ‘z’: -0.494384765625}, ‘ground_distance’: 0.4555206298828125, ‘speed’: {‘north’: 0.0, ‘east’: 0.0, ‘down’: 0.0}, ‘battery_percentage’: 47, ‘flying_state’: ‘FS_LANDED’}, ‘camera’: {‘timestamp’: 110000074, ‘base_quat’: {‘w’: 0.8690185546875, ‘x’: 0.0, ‘y’: 0.0, ‘z’: -0.49462890625}, ‘quat’: {‘w’: 0.8690185546875, ‘x’: 0.0, ‘y’: 0.0, ‘z’: -0.49462890625}, ‘exposure_time’: 30.0390625, ‘iso_gain’: 1118, ‘awb_r_gain’: 1.967529296875, ‘awb_b_gain’: 1.852294921875, ‘hfov’: 1.2042771577835083, ‘vfov’: 0.7377424836158752}, ‘links’: [{‘wifi’: {‘goodput’: 11017, ‘quality’: 5, ‘rssi’: -49}}]})

while in photo mode I get this:

(<VMetaFrameType.PROTO: 6>, {‘drone’: {‘quat’: {‘w’: 0.8695068359375, ‘x’: 0.02117919921875, ‘y’: 0.038818359375, ‘z’: -0.49188232421875}, ‘ground_distance’: 0.3879852294921875, ‘speed’: {‘north’: 0.0, ‘east’: 0.0, ‘down’: 0.0}, ‘battery_percentage’: 46, ‘flying_state’: ‘FS_LANDED’}, ‘camera’: {‘timestamp’: 149424533, ‘base_quat’: {‘w’: 0.870361328125, ‘x’: 0.0, ‘y’: 0.0, ‘z’: -0.4923095703125}, ‘quat’: {‘w’: 0.8702392578125, ‘x’: 0.0, ‘y’: 0.0, ‘z’: -0.49261474609375}, ‘exposure_time’: 30.0390625, ‘iso_gain’: 1071, ‘awb_r_gain’: 1.92138671875, ‘awb_b_gain’: 1.849609375, ‘hfov’: 1.3177236318588257, ‘vfov’: 1.05217444896698}, ‘links’: [{‘wifi’: {‘goodput’: 16905, ‘quality’: 5, ‘rssi’: -44}}]})

I cannot reproduce this difference, if I use API functions to switch between photo and video.

For both modes these readings came way closer to my calibration results then what I have reported above… Is there any other secret switch I don’t know about? Or is this a “special” photo mode you are setting in your app?

[Neil]

I have really difficulties to understand, could you please explain?

When I setup the drone via the Freeflight app to use photo mode and entering my app (w/o switching mode, zoom or whatever), I’m reading these values for width, height, hfov and vfov from the stream

1024 768 1.3177236318588257 1.05217444896698

If I switch to video mode, these settings change to

1280 720 1.2042771577835083 0.6954046487808228

If I repeat that photo/video switch in my app several times, then it might show the same results as above for one or two more attempts, but finally it lands here:

1280 720 1.2042771577835083 0.6954046487808228
1024 768 1.2042771577835083 0.6954046487808228

Can you explain me that?

Here my photo/video switch methods. What am I missing?

    def set_photo_mode(self):
        ''' Set photo mode '''
        if not self.drone.connection_state():
            return
        self.logger.info("set photo mode")

        self.drone(camera.set_camera_mode(cam_id=0, value='photo'))
        self.drone(camera.set_photo_mode(
                cam_id=0,
                mode='single',
                format='full_frame',
                file_format='jpeg',
                burst='burst_4_over_1s',
                bracketing='preset_1ev',
                capture_interval=0
        ))

        self.logger.info("photo mode set")


    def set_video_mode(self):
        ''' Set video mode '''
        if not self.drone.connection_state():
            return
        self.logger.info("set video mode")

        self.drone(camera.set_camera_mode(cam_id=0, value='recording'))
        self.photo_video_mode = 'video'
        self.logger.info("video mode set")

[ndessart]

We will investigate this live streaming metadata consistency issue.

Could you please reproduce it and share the logs from the drone (the logs.bin files that you can get from the drone web server http://192.168.42.1 when connected to your drone wifi SSID). Thanks

[Neil]

Ok, to recap what I did:

1. I used the Android Freeflight app in order to set the drone into photo mode (since it was already in photo mode I switched back to video and then to photo again)
2. I left the app
3. I started my app, not changing photo/video mode and zoom
4. I switched back and force between photo and video using the routines shown in my initial post regarding this
5. I observed width, height, hfov and vfov as reported in the stream meta data using the extraction methods shown in my initial post
6. I could confirm my observations reported in the initial post: While the drone is initially reporting different hfov and vfov values for photo and video mode, it later levels that and reports just the video values for both resolutions/modes.

• With photo mode set from the Freeflight app it starts with:

1024 768 1.3177236318588257 1.05217444896698

• I switch to video:

1280 720 1.2042771577835083 0.6954046487808228

• Back to photo:

1024 768 1.3177236318588257 1.05217444896698

• Back to video:

1280 720 1.2042771577835083 0.6954046487808228

• Back to photo:

1024 768 1.2042771577835083 0.6954046487808228

• From now on I get the same hfov/vfov for photo and video (well, sometimes with slight variations from the first or fourth digit after the decimal point, but generally the same). A restart of the script (reconnect to the drone) doesn’t change that. Only the repeated use of the FreeFlight app returns to the initially reported hfov/vfov values for photo mode, until it goes again to the same values while using my app.

I have a “bin” recording. Will try to attach it here, but it is 190 MB, I doubt it will work

EDIT: As thought, can’t upload it here. I will ask Daniel to forward you the dropbox link by mail

[ndessart]

Thanks for the logs received via private messages. We’re still investigating this issue.