[SAMPLE CODE] New Unified Version for all Drones!


#1

Dear Developers,
we are pleased :tada: to release a unified version of sample code for all Parrot Drones (Airborne, Jumping, Bebop1/2, SkyControler), on iOs and Android platform ! :gift:

You can download the samples here, and start coding amazing apps from those template.

Huge work from @Djavan !! :grinning:


Bebop 2: How to use the samples
Bebop2 video issues
#2

#3

Hi @Jerome and @Djavan,

I have been playing with the Unified example and I’ve ran into an interesting scenario with BebopVideoView and ARDeviceControllerStreamListener.

Check out this use case / chain of events:

Activity A performs discovery, finds a drone and initiates a connection through a service.

User chooses to pilot drone and launches Activity B (Pilot Activity) from Activity A.

Activity B interacts with service, registers itself as a stream listener and enables the video stream.

Activity B receives configureDecoder event and subsequent onFrameReady events and forwards them on to BebopVideoView.

The user then launches Activity C, let’s call this our Settings Activity from Activity B. onPause Activity B stops the video stream and unregisters itself as a stream listener.

The user does stuff in Activity C and leave the activity (back pressed). onResume fires on Activity B, it re-establishes its service connection, re-registers itself as a stream listener and enables the video stream. configureDecoder does not fire, but onFrameReady does. This is fine because the byte buffers for the decoder / codec are still valid / referenced as Activity B was never actually destroyed.

The user then decides that they want to exit Activity B and leave that activity, thus resuming Activity A destroying Activity B in the process. The user does stuff in Activity A and then decides to re-launch activity B. Activity B is recreated, rebinds to the underlying service that is maintaining the deviceController reference, registers itself as a stream listener and enables video. onFrameReady fires but because the decoder / codec was not initialized fails while attempting to process inside BebopVideoView’s displayFrame method.

Is there a way to re-initialize the decoder / codec inherently? I can move the actual stream listener into the service maintaining the deviceController connection but I don’t want to. I can also stuff csd-0 and csd-1 into a static or similar buffer but I don’t want to do this either.

Thoughts?


#4

Hi @synman,

By returning an error on the callback onFrameReceived, you’ll force the SDK to send you the configuration on the next frame.

To take the Samples as example, we could do the following changes:

  • notifyFrameReceived returns an ARCONTROLLER_ERROR_ENUM.
  • onFrameReceived returns an ARCONTROLLER_ERROR_ENUM.
  • displayFrame returns an ARCONTROLLER_ERROR_ENUM. If mMediaCodec is null, returns an error (for example ARCONTROLLER_ERROR_INIT_STREAM), otherwise, returns ARCONTROLLER_OK.

Would this solution fit your needs?

Best regards,
Djavan


#5

that would work perfectly! and would help in other cases as well!

I actually rolled out a new version of my app leveraging my service component to maintain the csd-0 and csd-1 buffers.

I like your recommendation quite a bit more as it gets my service out of the video streaming business :slight_smile:


#6

#7

Hi @Djavan,

I just want to confirm. The SDK supports this capability today?

If I return ARCONTROLLER_ERROR_INIT_STREAM in onFrameReceived it will always fire configureDecoder before firing another onFrameReceived?


#8

Hi @synman,

Yes it does, with any returned value except ARCONTROLLER_OK. :slight_smile:

Regards,
Djavan


#9

@Djavan does this approach (restarting the video stream in the event of an error) also apply to processing of Mambo (RTSP) frames?

I’m noticing some oddities (perhaps due to lack of maturity) of the video stream implementation for the Mambo FPV.


#10

Hi,

Please note that it does not restart the video stream, it just calls again the decoder configuration callback and generates a full grey image.

This should be correctly handled by libARStream2 even for the Mambo. What kind of oddities are you noticing?

Forcing the libARStream2 to generate this grey i-frame is needed on the Bebop 2 because it is using a periodic intra-refresh mechanism. This means that there is no i-frame on the Bebop 2, only frames with i-slices and p-slices. So the decoder does not have a starting reference frame. This is why we generates a fake i-frame.
The Mambo produces an i-frame each second or so, this is why you should be able to always return OK even if the decoder failed at some point. The decoder will work again when it will receive the next i-frame.


#11

I think the crux of the issues are within my implementation but the absence of a ARCONTROLLER_DICTIONARY_KEY_ARDRONE3_MEDIASTREAMINGSTATE_VIDEOENABLECHANGED has complicated things.

I notice in the samples that startVideoStream is blindly sent for mini drones and no unregister of the stream listener or stop command ever occurs.

I’m trying to follow this pattern as much as possible but because of behavioral differences I see stuff like the rtsp_client freaking out subsequently with constant connect / disconnect messages and general instability that require a full restart of the ARSDK lifecycle and reboot of the Mambo to recover from.


#12

Yeah, there is definitely something funky going on. I’ve tested with the prebuilts and my builds of the 3.13 SDK using a Mambo v3.0.17.

It appears to be tied to the ARDeviceController lifecycle. The Android sample sandboxes It within an Activity. If you move it into a service you’ll the see the rtsp client freak out and go into a never ending connect/disconnect loop.

12-21 22:57:22.371 15090-15235/.debug I/librtsp: client disconnected
12-21 22:57:23.132 15090-15420/.debug I/librtsp: client connected
12-21 22:57:23.134 15090-15511/.debug I/librtsp: client disconnected
12-21 22:57:24.419 15090-15235/.debug I/librtsp: client connected
12-21 22:57:24.458 15090-15420/.debug I/librtsp: client disconnected
12-21 22:57:25.149 15090-15511/.debug I/librtsp: client connected
12-21 22:57:25.154 15090-15235/.debug I/librtsp: client disconnected

The only way to get out of this state is to completely terminate the app. Toggling Start/Stop Video Stream has no effect.


#13

ok,

I was able to work around it by doing an explicit dispose() of the deviceController once the DISCONNECT event fires after calling stop().

I still have an issue maintaining the session across activities but will dig a little deeper now that I’ve got a work-around for the lifecycle in place.

This appears to be tied to the magical events that clean up librtsp in dispose():

12-21 23:33:48.725 17939-18245/.debug E/pomp: fd=147, cb=0x9edd15f1 not removed from loop
12-21 23:33:48.726 17939-18245/.debug E/pomp: fd=145, cb=0x8d902505 not removed from loop
12-21 23:33:48.726 17939-18245/.debug E/pomp: fd=142, cb=0x8d901ff9 not removed from loop

Ideally the deviceController lifecycle is consistent across drone types (please) :slight_smile:


#14

Wanted to recap my observations:

  1. deviceController does not inherently clean up librtsp and libpomp references on stop(). I’m actually okay with this as in hindsight I should’ve been calling dispose anyway (otherwise Parrot wouldn’t have made it public).

  2. Mambo video stream is only valid as a “one shot” per deviceController session. You are currently unable to stop and restart the RTSP stream within a single lifecycle of the deviceController. It would be really nice to see this addressed – and possibly add a Video Stream State event to go along with it.