Reading Frame Metadata From a YUV Sink in Java

[mikekit]

I am trying to read the frame metadata (Embedded Video MetaData) using a YUV sink in Java. I am interested in the metadata_v3_base structure (72 bytes). I show my attempt below but does anyone have a Java example on how to do this?

The following two lines start the sink from the code that starts the stream view.

YUVSink.Config sinkConfig = YUVSink.config(Looper.getMainLooper(), cb);
liveStream.openSink(sinkConfig);

Here’s the callback from the YUVSink. I’m focused on the onFrame callback.

YUVSink.Callback cb = new YUVSink.Callback() {

    @Override
    public void onStart(@NonNull YUVSink sink) {
    }

    @Override
    public void onFrame(@NonNull YUVSink sink, @NonNull YUVSink.Frame frame) {
        Pointer ptr = new Pointer(frame.nativePtr());
        System.out.printf("rDebug onFrame 0x%2x 0x%2x\n", ptr.getByte(0), ptr.getByte(1));
        frame.release();
    }

    @Override
    public void onStop(@NonNull YUVSink sink) {
    }
};

The onFrame is being called for each video frame but I’m having trouble figuring out what to do with the native pointer returned by frame.nativePtr(). I’m using the Pointer object right now. The metadata_v3_base structure should have 0x50 and 0x30 (“P3”) as the first two bytes but I never see them. I’m new to the Anafi SDK so my approach to reading video frame data in Java might be completely wrong.

I’m referencing https://developer.parrot.com/docs/pdraw/metadata.html for the metadata_v3_base structure.

Thank you and all the best.

[mvinel]

Hi,

YUVSink is an internal API that has not yet been stabilized and considered practical enough to be made public. So if you use this API, be aware that it is unstable and that it may change in the future.

That said, the Frame.nativePtr() method will return a pointer to the native frame object; however, this is not a metadata_v3_base in itself, but a struct sdkcore_frame (declared sdkcore/src/main/jni/stream/include/sdkcore/sdkcore_frame.h).

To access the frame metadata, you would need to use function sdkcore_frame_get_metadata in sdkcore/src/main/jni/stream/include/sdkcore/internal/sdkcore_frame.h.
Here again, be aware that those are internal APIs.

I am afraid we do not provide Java APIs to access frame metadata as you would like to do; you will have to resort to native/JNI code for that.

I suppose the ‘Pointer’ class you use allows you to access native memory using the returned pointer, and from there, you may be able to get to the metadata pointer using only java and following the definition of all nested structures in sdkcore frame, but that may be a bit harsh, and you also need to refer to the source code of other native libraries that constitute the SDK, such as PDRAW, libvideo_buffers, libvideo-metadata, and so forth.

Hope that answers your question.

Regards.

[cstallings]

@mvinel, Thank you for your reply to @mikekit’s question. I have a more basic followup question.

Is there any approach currently that would allow you to extract frame metadata from a stream, using format 3 on Android? The documentation indicates that you may be able to do that either using YUV or H.264.
If yes, my followup question would be, can you provide an example of that?

Thank you in advance for your help.

[mvinel]

Hi,

No there is no documented approach or examples of how to do that on Android.
The only possible solution for now is to rely on internal native APIs as described above.

Regards.