Accessing vmeta_frame data arriving in ffmpeg

ANAFI USA / ANAFI Thermal / ANAFI 4K
1

Hi. We have an existing app that relies upon FFMPEG for video stream demuxing and decoding. It sees the parrot: Duration: 00:02:35.39, start: 0.000000, bitrate: 60574 kb/s
Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 59997 kb/s, 59.95 fps, 59.94 tbr, 60k tbn, 119.88 tbc (default)

  • Metadata:*
  •  creation_time   : 2020-10-02T17:14:45.000000Z*

  •  handler_name    : DefaultVideo*
  • Stream #0:1(eng): Data: none (mett / 0x7474656D), 54 kb/s (default)*
  • Metadata:*
  •  creation_time   : 2020-10-02T17:14:45.000000Z*

  •  handler_name    : ParrotVideoMetadata*
  • Stream #0:2(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 384 kb/s (default)*
  • Metadata:*
  •  creation_time   : 2020-10-02T17:14:45.000000Z*

  •  handler_name    : DefaultAudio*
  • Stream #0:3: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 320x180 [SAR 1:1 DAR 16:9], 90k tbr, 90k tbn, 90k tbc*

My question is What is the most efficient way for me to access the vmeta_frame structure, so I can operate on it with libvideo-metadata to get out the metadata of interest?

I have used this libvideo-metadata with pdraw_be - and it works perfectly - but it seems to me that since I already see the metadata frames, I should be able to access the vmeta_frames directly.

Thank you in advance for your help.

Mike Abernathy

3 
   while (av_read_frame (afc, p) >= 0)
   {

      if (p->stream_index == metadatastream)
      {
			printf("metadata packet of size %d\n", p->size);

			struct vmeta_buffer buf;
			
			vmeta_buffer_set_cdata(&buf, p->data, p->size,0);

			struct vmeta_frame meta;
			vmeta_frame_read(&buf, &meta, NULL);
			
			struct vmeta_location loc;
			vmeta_frame_get_location(&meta, &loc);
			
			printf("location latitude %f longitude %f altitude %f\n", loc.latitude, loc.longitude, loc.altitude);
4

That is exactly what I needed.

Thank you very, very much!

Mike Abernathy

5

Screen Shot 2020-10-23 at 3.55.15 PM
Screen Shot 2020-10-23 at 3.55.15 PM2784Ă—1828 1.7 MB

This is our LandForm Studio app running on a Mac. It is an Augmented Reality display that overlays street lines, street names and place marks on the drone video in real-time. Thank you again for your help.

3 Likes
6

This appears to work just fine for MP4 files with Parrot metadata in the data stream as above. But, for an RTSP connection (which FFMPEG supports), I believe we are just getting video packets, and the metadata is instead stuffed into some RTP header extensions. How do we access them in FFMPEG after the “av_read_frame” ?

7

I am looking to do the same. Is there a way to write frame metadata into an MP4 file by using RTSP stream as input? I have tried several FFMPEG command line options without any luck. If I use an MP4 from Parrot SD card, all of this is there. I can also capture the packets in a .pacp file and see the metadata and extract it. But I am unable to do this using FFMPEG reading the RTSP stream and writing output to an MP4. Any help is much appreciated. Is this possible at all?

8

Sorry for the late reply, I do not check this forum every day.

Getting the metadata form the RTSP stream using ffmpeg is possible. Not from the command line though.

I will post an example on Tuesday.

1 Like
9 
   while (av_read_frame (afc, p) >= 0)
   {
		int extlen = p->extlen;
		if (extlen == 84)
		{
			uint32_t* telemetry = (uint32_t*) p->ext;
10

This indeed would be absolutely perfect as I only need an FFMPEG development solution not from the command line. However, I’m using the recent FFMPEG 4.3.1 baseline and I do indeed have packets coming in through av_read_frame. However, my AVPacket does NOT have an ext or extlen member. Do you mean the “AVPacketSideData *side_data;” member?

11

ok, it’s been some time ago I made this. Then I probably patched ffmpeg to expose the ext and extlen. Let me check…

Diff says I made changes to:
libavformat/rtpdec.c
libavformat/utils.c
libavcodec/avpacket.c
libavcodec/avcodec.h

How can I give you the changed files?

12

Can you post the files on a GitHub or in an archive ?
thanks!

1 Like
13

Any public way would be great (e.g. Github, Google Drive zip file, etc). Also, I wrote you a direct message with my email address. I think many folks would benefit from this code and it should probably be kept up-to-date with the latest FFMPEG baseline and generally be made available to folks on your developer forum. Most developers writing apps to support video streaming are already using FFMPEG, so a solution using for it would be best.

PDraw could be a solution, but would be a nightmare for us right now. It’s not just the logistics of reworking the PDraw library AND all of its dependencies so they work in windows for the compiler of our choice. It’s also that an exception would have to be made in our FFMPEG based programs at each step of the way to handle this single case: Parrot metadata. That is, for just Parrot metadata, redirection to your libraries would have to be made during connection initialization as well as every place packets are consumed, and finally at closing time. We use the FFMPEG libraries for both file based streams as well as network streams. FFMPEG supports something like 30 different streaming protocols. It is cross platform, and even compiles nicely as a VCPKG for the Visual Studio windows folks. We also use FFMPEG for it’s other API, e.g. (seek, encoding, queries, etc…) as well as encode/decode support. FFMPEG does indeed provide RTSP support as you can see. It just appears to ignore this rarely used optional header extension, and skips over it.

14

hsteinberg, i sent you the files using email.

1 Like