Grab frame off MediaCodec and pass it to SurfaceView


#1

Hey there,

i’m currently developing an autonomous flight app (Java/Android), which shall display the video stream and, on top, grab the frames to pass it to ZXing + OpenCV to detect QR-Codes and measure the distance.

I’m currently using a MediaCodec to decode the frames, which works as long as i pass my SurfaceView directly to the codec to draw it…
But when doing so, i cannot grab the actual frame, so i need to get the OutputBuffers manually (mediaCodec.getOutputBuffer(outIndex)) and feed the View + hand the data to ZXing + OpenCV.

So, my question is, how do i manually feed the SurfaceView and make my application draw it onto the display?

This is my current onFrameReceived:

public ARCONTROLLER_ERROR_ENUM onFrameReceived(ARDeviceController deviceController, ARFrame frame)
{
    readyLock.lock();

    if ((mediaCodec != null))
    {
        if (isCodecConfigured) {
            // Here we have either a good PFrame, or an IFrame
            int index = -1;

            try
            {
                index = mediaCodec.dequeueInputBuffer(VIDEO_DEQUEUE_TIMEOUT);
            }
            catch (IllegalStateException e)
            {
                Log.e(TAG, "Error while dequeue input buffer");
            }
            if (index >= 0)
            {
                ByteBuffer b = buffers[index];
                b.clear();
                b.put(frame.getByteData(), 0, frame.getDataSize());

                try
                {
                    mediaCodec.queueInputBuffer(index, 0, frame.getDataSize(), 0, 0);
                }
                catch (IllegalStateException e)
                {
                    Log.e(TAG, "Error while queue input buffer");
                }
            }
        }

        // Try to display previous frame
        MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
        int outIndex = -1;
        try
        {
            outIndex = mediaCodec.dequeueOutputBuffer(info, 0);

            while (outIndex >= 0)
            {
                mediaCodec.releaseOutputBuffer(outIndex, false);
                outIndex = mediaCodec.dequeueOutputBuffer(info, 0);

            }
        }
        catch (IllegalStateException e)
        {
            Log.e(TAG, "Error while dequeue input buffer (outIndex)");
        }
    }

    readyLock.unlock();


    return ARCONTROLLER_ERROR_ENUM.ARCONTROLLER_OK;
}

Cheers


#2

I’m not sure (this is more an Android question than a drone SDK one), but I think you have to use an OpenGL texture to do that.
You’ll find more information here: https://developer.android.com/reference/android/media/MediaCodec.html