Hello everyone,
I am currently trying to implement something very similar to what @DHBW mentioned in this post:
I’m trying to display the video stream from a Parrot Anafi in an OpenCV window and/or do some “real-time calculations” for the live images. I haven’t worked that much with Python yet and was very grateful for the information in the
installation.rst (Olympe v7.0.2).
After the installation I adapted the Streaming.py example as I thought it might work.
I have also uploaded this modified version to my GitHub for you.
It seems to work…, but only for about 150 seconds. After the 150 seconds my virtual machine freezes and after a short time the program is killed. The kernel buffer indicates that the process was killed because there is no free memory available.
Maybe it has something to do with Olympe itself, or maybe my implementation is wrong. In any case, I would appreciate any help to solve the problem.
Below I have sort of the minimum viable product to get the video stream from Olympe displaying in a cv2 window. I adapted it from our streaming code that we have working. I tested this in sphinx using v7.0.0 of olympe. I ran it for 5 minutes and looked at the memory utilization on the system. While the total load was close to the total of 16GB of RAM on my laptop, I didn’t see a consistent increase as in the memory usage for the pimp-video process which seems to be the process that grabs the frames from the drone. The allocated memory seemed to fluctuate normally over the 5 minute test. Note, we are only using the yuv frames and are not handling the h264 callback.
Hope this helps you locate the issue in your VM. One thing to try would be to increase the memory appropriated to your VM from the hypervisor. For example, if you double from 8GB to 16GB, does the program run twice as long before OOM? If so, then you probably are leaking memory.
Regards,
Tom
import olympe
from olympe.messages.ardrone3.Piloting import TakeOff, moveBy, Landing, moveTo, NavigateHome
import threading
import time
import queue
import cv2
import logging
class OlympeStreaming(threading.Thread):
def __init__(self, drone):
self.drone = drone
self.frame_queue = queue.Queue()
self.flush_queue_lock = threading.Lock()
self.frame_num = 0
self.renderer = None
super().__init__()
super().start()
def start(self):
# Setup your callback functions to do some live video processing
self.drone.streaming.set_callbacks(
raw_cb=self.yuv_frame_cb,
h264_cb=self.h264_frame_cb,
start_cb=self.start_cb,
end_cb=self.end_cb,
flush_raw_cb=self.flush_cb,
)
# Start video streaming
self.drone.streaming.start()
#self.renderer = PdrawRenderer(pdraw=self.drone.streaming)
def stop(self):
if self.renderer is not None:
self.renderer.stop()
# Properly stop the video stream and disconnect
self.drone.streaming.stop()
def yuv_frame_cb(self, yuv_frame):
"""
This function will be called by Olympe for each decoded YUV frame.
:type yuv_frame: olympe.VideoFrame
"""
yuv_frame.ref()
self.frame_queue.put_nowait(yuv_frame)
def flush_cb(self, stream):
if stream["vdef_format"] != olympe.VDEF_I420:
return True
with self.flush_queue_lock:
while not self.frame_queue.empty():
self.frame_queue.get_nowait().unref()
return True
def start_cb(self):
pass
def end_cb(self):
pass
def h264_frame_cb(self, h264_frame):
pass
def display_frame(self, yuv_frame):
# the VideoFrame.info() dictionary contains some useful information
# such as the video resolution
info = yuv_frame.info()
height, width = ( # noqa
info["raw"]["frame"]["info"]["height"],
info["raw"]["frame"]["info"]["width"],
)
# yuv_frame.vmeta() returns a dictionary that contains additional
# metadata from the drone (GPS coordinates, battery percentage, ...)
# convert pdraw YUV flag to OpenCV YUV flag
cv2_cvt_color_flag = {
olympe.VDEF_I420: cv2.COLOR_YUV2BGR_I420,
olympe.VDEF_NV12: cv2.COLOR_YUV2BGR_NV12,
}[yuv_frame.format()]
# yuv_frame.as_ndarray() is a 2D numpy array with the proper "shape"
# i.e (3 * height / 2, width) because it's a YUV I420 or NV12 frame
# Use OpenCV to convert the yuv frame to RGB
cv2frame = cv2.cvtColor(yuv_frame.as_ndarray(), cv2_cvt_color_flag)
cv2.imshow("Frames via Olympe", cv2frame)
cv2.waitKey(1)
def run(self):
main_thread = next(
filter(lambda t: t.name == "MainThread", threading.enumerate())
)
while main_thread.is_alive():
with self.flush_queue_lock:
try:
yuv_frame = self.frame_queue.get(timeout=0.01)
except queue.Empty:
continue
try:
self.display_frame(yuv_frame)
except Exception as e:
print(e)
finally:
# Don't forget to unref the yuv frame. We don't want to
# starve the video buffer pool
yuv_frame.unref()
logger = logging.getLogger(__name__)
if __name__ == "__main__":
#eventually IP will be specified depending on what drone is chosen
IP = "10.202.0.1"
drone = olympe.Drone(IP)
drone.connect()
drone(TakeOff()).wait().success()
streamer = OlympeStreaming(drone)
streamer.start()
### Flight commands here ###
time.sleep(300)
streamer.stop()
drone(Landing()).wait().success()
drone.disconnect()
Hi Tom,
first of all, thanks for the quick answer.
I took your script and ran it on my virtual machine. Unfortunately, I got the same result with your script. The process works but is killed after about 170 seconds with the reason that there is no more memory available. I have a Windows machine with a total of 8GB RAM and the virtual machine is allowed to reserve 5GB. Ignoring the fact that Windows already reserves almost half of this and only 4.6GB is available.
However, I don’t use Sphinx either but connect to a physical drone. Does the process then actually reserve so much memory over time that the process has to be killed?
Or do I really have to buy a computer with more RAM for the project…?
It sounds like the VM might be allocated more memory than the Windows host can really supply it. I believe you can limit the memory of the process using Linux cgroups and executing the olympe script within the control group. However a better option might be to use the Linux subsystem in Windows 10 so that you can install Ubuntu natively. You then won’t have any of the overhead of the VM and should be able to use all of the available 8GB RAM. You can follow these steps here to install WSL and Ubuntu: Install Ubuntu on Windows 10 | Ubuntu
Hi @teiszler,
I borrowed a computer with 32GB RAM from my university. I set up everything (Ubuntu, Olympe…) on it and ran your script again. I allocated 30GB to the virtual machine. The same problem…After 550 seconds, the process consumes 25GB of RAM!!!, after the RAM is completely used, the operating system starts to swap memory to the hard disk (another 3 gigabytes). As soon as this memory pool is used up, the process is killed. I have also used gc.collect() to see if I have any memory leaks coming from Python, but that didn’t help either. So either it’s my/your programme, one of the current Python libraries, or the C buffers that Olympe creates internally. But I don’t know what else I can do.
The main difference I see is that you are still running it in a VM. Was the 32GB machine also a Windows 10 host? I would try removing virtualization from the equation and install WSL and Ubuntu natively on Windows using the instructions in my post above. Alternatively you could try a bare metal installation of Ubuntu which is what I am using.
Hi @teiszler,
I have now borrowed another laptop today, installed Ubuntu on the bare metal. The laptop has 16GB of RAM, which filled up after about 350 seconds. I tested my program and yours again. I got the same result with both programs. After excluding that it has something to do with the virtual machine, only my/your implementation, a problem in a Python library or a problem with Olympe remains.
I am at a loss and would appreciate any help!
Are you certain it is the Olympe script that is the problem? Are you running in the simulator? If not, can you try it on a real drone and see if the same thing happens? Is it the older Gazebo 7-based simulator or the new UE4 version? How did you install Olympe? My only guess is that it has something to do with the way you setup your environment. Or you genuinely found a memory leak in the streaming libraries, though I would imagine if that were the case, there would be more people encountering this issue. For what is it worth, I am running Olympe 7.0.0 with the old 1.8 sphinx simulator.
sphinx --version
Parrot-Sphinx simulator version 1.8
Gazebo multi-robot simulator, version 7.0.1
Copyright (C) 2012-2015 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org
Gazebo multi-robot simulator, version 7.0.1
Copyright (C) 2012-2015 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org
@teiszler,
I don’t use a simulator at the moment, I always connect to a physical drone (Anafi 4K). So it can’t be due to a simulator either. I installed Olympe after the ‘Installation.rst’. I just installed python and pip as a global setup, created a virtual python environment and activated it. In this virtual environment I then upgraded pip, installed Pysdl2, PyOpenGL and OpenCv. After that I installed Olympe (7.0.2) via pip into the virtual environment, because a manual installation still seems to be impossible despite version 7.0.2, because postinst as well as the \env folder still seem to be missing (or I just can’t find them). But I can’t think of any other cause than the Python script. Either I implemented it wrong, but after running into the same error with your script I don’t know what I could have done wrong . Or it has something to do with a library.
I think that this memory leak is due to a bug in the olympe “streaming.py” example.
Here:
The yuv_frame_cb enqueue a frame (after incrementing its reference count) into a queue that is never processed. The user is supposed to process the YUV frames asynchronously from another thread and is responsible of unreferencing the frames there. This part is completely missing from the example and must have been deleted inadvertently with the olympe 7.0 release. I’ll update the example for the next release. Sorry for the inconvenience.
After further investigations, there is indeed a remaining memory leak within the video frame object.
The following patch on Olympe 7.0.3 should fix this issue.
diff --git a/src/olympe/video/frame.py b/src/olympe/video/frame.py
index 0f49584..60acbbf 100644
--- a/src/olympe/video/frame.py
+++ b/src/olympe/video/frame.py
@@ -176,6 +176,10 @@ class VideoFrame:
if res < 0:
self.logger.error(f"mbuf_raw_video_frame_finalize returned error {res}")
return self._packed_video_frame
+ # Let the frame own the last ref count on this buffer
+ res = od.mbuf_mem_unref(self._packed_buffer)
+ if res < 0:
+ self.logger.error(f"mbuf_mem_unref returned error {res}")
return self._packed_video_frame
def as_ctypes_pointer(self):
Could you please confirm this fix your issue so that I can include this patch in the next Olympe release ?
Thank you
I have also come to the conclusion that it must be this memory. But after trying almost exactly the same thing yesterday, I was about to give up.
As it turns out, I just tried it in the wrong place.
It fixes the problem. Thank you so much for looking at this so quickly. It made my day!!
With kind regards
Sebastian