Turn on verbose logging for Olympe, then search for “camera.photo_capabilities” in olympe.log. You must use one of the format combinations listed there or else your set_photo_mode command will fail. Here is an example of what I see in olympe.log on my drone:
_recv_cmd_cb - camera.photo_capabilities(id=0, photo_modes='single|time_lapse|gps_lapse', photo_formats='full_frame', photo_file_formats='dng_jpeg', hdr=supported.not_supported, list_flags='First')
_recv_cmd_cb - camera.photo_capabilities(id=1, photo_modes='single|time_lapse|gps_lapse', photo_formats='rectilinear', photo_file_formats='dng_jpeg', hdr=supported.not_supported, list_flags='')
_recv_cmd_cb - camera.photo_capabilities(id=2, photo_modes='single|burst|time_lapse|gps_lapse', photo_formats='full_frame', photo_file_formats='jpeg', hdr=supported.supported, list_flags='')
_recv_cmd_cb - camera.photo_capabilities(id=3, photo_modes='single|burst|time_lapse|gps_lapse', photo_formats='rectilinear', photo_file_formats='jpeg', hdr=supported.supported, list_flags='')
_recv_cmd_cb - camera.photo_capabilities(id=4, photo_modes='bracketing', photo_formats='full_frame', photo_file_formats='jpeg|dng_jpeg', hdr=supported.not_supported, list_flags='')
_recv_cmd_cb - camera.photo_capabilities(id=5, photo_modes='bracketing', photo_formats='rectilinear', photo_file_formats='jpeg|dng_jpeg', hdr=supported.not_supported, list_flags='Last')
DNG is only supported for some of these entries. After you call set_photo_mode, what shows up in olympe.log? You should see a photo_mode() response log entry indicating whether it worked or not. See docs
_send_command_impl - camera.set_photo_mode(0, <photo_mode.single: 0>, <photo_format.rectilinear: 1>, <photo_file_format.jpeg: 0>, <burst_value.burst_14_over_1s: 2>, <bracketing_preset.preset_1ev: 0>, 0.0) has been sent to the device
...
_recv_cmd_cb - camera.photo_mode(cam_id=0, mode=photo_mode.single, format=photo_format.rectilinear, file_format=photo_file_format.jpeg, burst=burst_value.burst_14_over_1s, bracketing=bracketing_preset.preset_1ev, capture_interval=0.0, list_flags='')