Olympe 7.3 leaking memory on failing drone.connect()

ANAFI Ai
1

My app tries to establish a connection with a real drone once per 5 secs.

        if not self.is_drone_connected():
            try:
                self.connection_lock.acquire()
                self.drone = olympe.Drone(self.args.anafi_drone_ip)
                if self.drone.connect():
                    self.logger.info("drone connected")
                else:
                    self.logger.error("drone NOT connected") 
                self.connection_lock.release() 
            except Exception as e:
                self.logger.error(f"exception on drone_connection_management: {e}, exit required")
                os._exit(0)
            finally:
                if self.connection_lock.locked():
                    self.connection_lock.release()

This runs well for pretty much exactly 7 minutes:

2022-09-02 19:18:27,936 [ERROR] olympe.drone - _do_connect - '192.168.42.1 connection retries failed
2022-09-02 19:18:34,192 [ERROR] olympe.drone - connect - '192.168.42.1 connection timed out
2022-09-02 19:18:34,195 [ERROR] parrot: drone NOT connected

After this time of unsuccessful attempts I see the memory consumption going up in htop (about 1MB per attempt).

Finally the SDK excepts (after spitting Too many open files and excepts with Unable to create pomp timer:

2022-09-02 19:25:34,195 [ERROR] 	ulog - pomp - eventfd err=24(Too many open files)
ERROR:root:Unhandled exception: Traceback (most recent call last):
  File "</home/pi/anafi-pi/./main.py>", line 3, in <module>
  File "<main>", line 1121, in <module>
  File "<parrot>", line 205, in drone_connection_management
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/mixins/streaming.py", line 36, in __init__
    super().__init__(*args, **kwds)
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/mixins/mission.py", line 36, in __init__
    super().__init__(ip_addr, *args, name=name, **kwds)
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/mixins/media.py", line 37, in __init__
    super().__init__(*args, **kwds)
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/arsdkng/controller.py", line 105, in __init__
    super().__init__(
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/arsdkng/cmd_itf.py", line 216, in __init__
    self._scheduler = Scheduler(self._thread_loop, name=self._name)
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/scheduler.py", line 421, in __init__
    super().__init__(DefaultScheduler(*args, **kwds))
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/scheduler.py", line 113, in __init__
    self._attr.default.subscribers_thread_loop = Loop(
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/concurrent/__init__.py", line 131, in __init__
    self._task_timer = self.create_timer(self._task_timer_cb)
  File "/home/pi/code/parrot-groundsdk/out/olympe-linux/staging/usr/lib/python/site-packages/olympe/concurrent/__init__.py", line 626, in create_timer
    raise RuntimeError("Unable to create pomp timer")
RuntimeError: Unable to create pomp timer

This is reproducible. Only chance to automatically survive this is leaving the app and restarting via an outer bash script.

2

Hi Neil,

It seems here that the self.drone object is leaked as this attribute is being overwritten here :

You should probably call self.drone.destroy() if you do not plan to use that instance anymore.

Alternatively, you may be interested to use the the retry Drone.connect() parameter (that defaults to 1).

3

Ah, this makes sense. I somehow missed this function. Will try. On vacation currently. Thanks also for the other answers I got today. Will respond inline.

4

The call to the undocumented drone.destroy() didn’t help. It now just takes longer until the RuntimeException raises. 20 minutes instead of 7 minutes before. The memory consumption is increasing and finally the number of open files is too high and a RuntimeException is thrown by the SDK.

Altered code:

        if not self.is_drone_connected():
            try:
                self.connection_lock.acquire()
                if self.drone is not None:
                    self.drone.destroy()
                    self.drone = None
                self.drone = olympe.Drone(self.args.anafi_drone_ip)
                if self.drone.connect():
                    self.logger.info("drone connected")
                else:
                    self.logger.error("drone NOT connected") 
                self.connection_lock.release() 
            except Exception as e:
                self.logger.error(f"exception on drone_connection_management: {e}, exit required")
                os._exit(0)
            finally:
                if self.connection_lock.locked():
                    self.connection_lock.release()


2022-09-07 11:17:17,359 [ERROR] parrot.py: drone NOT connected
2022-09-07 11:17:17,358 [ERROR] 	olympe.drone - connect - '192.168.42.1 connection timed out
2022-09-07 11:17:17,368 [ERROR] 	olympe.drone - _do_connect - '192.168.42.1 connection retries failed

...

2022-09-07 11:37:53,177 [ERROR] parrot.py: drone NOT connected
2022-09-07 11:37:53,176 [ERROR] 	olympe.drone - connect - '192.168.42.1 connection timed out
2022-09-07 11:37:53,186 [ERROR] 	olympe.drone - _do_connect - '192.168.42.1 connection retries failed
2022-09-07 11:37:53,448 [ERROR] 	ulog - pomp - eventfd err=24(Too many open files)
2022-09-07 11:37:53,450 [ERROR] parrot.py: exception on drone_connection_management: Unable to create pomp timer, exit required

I have commented the entire sequence above and did let it run for over an hour. The memory consumption was rock solid at around 335 MB of 1.63 GB and just moved slightly up and down, depending on the other activities of the app.

In contrary the memory consumption goes up until about 410 MB and then the RuntimeException comes, if I re-enable the connect attempts.

So I’m quite sure there is something inside drone.connect() which consumes the memory and doesn’t release it on fail. Or my usage is completely wrong.

BTW: The SDK error traces appear after my app has the result of the operation. Strange?

5

I’m able to reproduce the leak with the following minimal script:

import olympe

drone = None

for i in range(1000):
    if drone is not None:
        drone.destroy()
        drone = None
    drone = olympe.Drone("10.202.0.1")
    if drone.connect():
        print("drone connected")
    else:
        print("drone NOT connected")

We don’t have the use cases of creating multiple Drone objects in the same script. We usually have create just one instance per script connecting multiple times to the same drone (if necessary). Anyway, I’ll investigate this issue.

In the meantime, would it be possible to instantiate just one instance of the Drone class ? This should be possible if you’re connecting to the same drone IP address.

6

Thanks for confirmation. I guess I will be able to change my script that way. Let me check and report

8

This topic was automatically closed after 30 days. New replies are no longer allowed.