Is there a gstreamer plugin to do frame metadata demuxing?

ANAFI USA / ANAFI Thermal / ANAFI 4K
1

Existing situation:
Parrot Anafi USA.
Using C++.
Simplified example pipelines to get video frames from mp4 or rtsp into our app:
filesrc location=intput_file.mp4 ! qtdemux ! h264parse ! queue ! avdec_h264 ! videoconvert ! appsink

rtspsrc location="rtsp://localhost:8554/test" ! rtph264depay ! h264parse ! queue ! avdec_h264 ! videoconvert ! appsink

Then using opencv to get the actual frames.
cv::VideoCapture(pipe_string, cv::CAP_GSTREAMER)

Question:
What would be the best way to extract live metadata out of a stream while still getting the frames at the same time?

  • Is there a gstreamer plugin that can do this?
    Something similar to qtdemux that has can separate video/audio/subtitles?

  • Or would it make sense to have the pipeline split in two using a tee leading to two appsinks, each handled separately?

  • Or would it be best to run something before the pipeline that separartes the frames and metadata and then pass those to our own app?

I have installed and had a play around with the PDrAW SDK.
I was able to use the native wrapper to output an mp4’s frame metadata to a csv file, but I am lacking the skills in C to take it further than that (ie. extract metadata during runtime of our app)

Regards,
Aron Thor

1 Like
2

Hello,

The official supported way to do this is indeed to use PDrAW (see Ground SDK Tools).
A helper library, libpdraw-vsink, is provided and implements an easy to use frame grabber that works on both recorded (MP4) and streamed (RTSP) videos, and outputs frame metadata along with the frames. See the test program for an example.

Doing the same with gstreamer would require developing a gstreamer plugin overriding the RTP depayloader to retrieve RTP extension headers and deserialize the metadata (using Parrot’s libvideo-metadata or a compatible reimplementation).

Regards

3

Thank you for your answer Akaaba.

While I’m working on that, can you recommend how I then get the raw frame data from it’s struct mbuf_raw_video_frame to a gstreamer input pipeline.
Failing that, perhaps pass it directly to opencv’s cv::Mat .

Regards

4

Have a look at libpdraw-vsink’s test program here.

You can use vdef_get_raw_frame_plane_count to get the plane count according to the YUV format, and then mbuf_raw_video_frame_get_plane / mbuf_raw_video_frame_release_plane to access each of the YUV planes.

The planes can then be wrapped or copied to another object for processing with OpenCV for example.