Sphinx Bugs and Flightplan Issues

Hello all,
My team and I have had quite the adventure trying to make flight plans work with the Anafi, both within Sphinx and in the real world. Below is a write-up of the issues we encountered and how we were able to solve/work around them (or not). I’m not sure if this would be more useful as a bug report on Github and whether it would be better as one large report or broken out into its components. It could be the basis of some new documentation, or at the very least, hopefully it helps anyone reading on the forum. Happy to provide more information on any of the topics!

QGroundControl
First, I should start by mentioning that we primarily relied on QGroundControl to make our flight plans. We realize this likely goes against the recommended best practices, but since we are trying to fly a grid pattern of an area at a consistent altitude with the camera facing nadir, it was the tool that made the most sense for the job. QGroundControl outputs its data in JSON format with a .plan extension, so we wrote a parser to convert it to QGC WPL 120 format (it was unclear whether 110 or 120 was preferred, but we had success with 120). We are happy to share the code once we have documented/cleaned it up a bit.

MAVLink Discrepancies
It is important to note a few discrepancies between standard MAVLink and the customized Parrot commands. For anyone following along at home, the Parrot command documentation is available here: https://developer.parrot.com/docs/mavlink-flightplan/messages.html. Parrot does not support a null/NaN option in the parameters – everything must be a float. This means that command 16 (waypoint)has no functionality to direct the yaw angle (for parameter 4) automatically.
The MAVLink documentation states that we should set parameter 4 to “NaN to use the current system yaw heading mode (e.g. yaw towards next waypoint, yaw to home, etc.).” (source: https://github.com/mavlink/mavlink/blob/master/message_definitions/v1.0/common.xml#1260).

We also discovered that it is not possible to generate a flight plan that has no flying actions. We wanted to generate a flight plan that only controlled the gimbal and took images to help reduce the risk of flying in a lab environment, but found that the flight plan parser in the drone expects valid waypoints before it will run the plan. Additionally, we found that issuing an explicit 22 (launch) command did not seem to be required. We were unsure whether this was intentional on Parrot’s part, but we preferred to add one anyways. We also had issues making the 20 (return to home) command work – perhaps the home location needs to be explicitly set within Freeflight?

Capturing Images
We noticed a lot of inconsistent behavior in our initial image capture tests in sphinx – sometimes all of the images were available, sometimes none were, and sometimes only some of the images were available. There were a number of reasons we experienced these issues.
We ended up using a real Anafi to help understand how to properly take images with consistent results flying flight plans in real life. First, we issued a 50001 (capture mode) command with parameters 1&2 set at 1.0 and 2.0, then issued 2000 (image) commands with the parameters 1-3 set at 0.0, 1.0, and 12.0, respectively. This wasn’t documented anywhere (or its location was unclear) and took some trial and error to discover.

Once we were able to take pictures properly in the real world, we found using this method plus setting the timestamp properly in Sphinx allowed us to consistently take images within Sphinx. The timestamp and the related Sphinx issues are detailed in the next section. However, we’ve had issues getting the simple front cam to work, and the images that it does capture are different dimensions (1280 x 720) than the images captured by the real drone (4608 x 3456).

Setting the Timestamp in Sphinx
Not having the timestamp set properly seemed to create a cascade of problems within Sphinx. Even when issuing image commands properly (mentioned in the above section), we had issues with the virtual SD card showing up as full and not writing images. This likely contributed to the inconsistent image capture behavior.
To set the simulation timestamp, we passed “–time=date +%s” as a command line argument when running Sphinx. This seemed to solve a number of our problems related to image capture, we also found that disconnecting and reconnecting the Freeflight app caused the SD card to show as full in Sphinx.
With the timestamp fix applied, the SD card still shows as full in Freeflight, but the images are captured anyways.

No Logs Available
Per our understanding, Sphinx and the Anafi’s operating system was initially based on Android. However Parrot’s logging system (ulog) differs heavily from Android. Sphinx ships with a full VM of a Parrot drone, but relies on some host services being present to make logging work. It would seem that the simulated Anafi drone is trying to write logs to /dev/ulog_main which is not a valid device in the simulator (we receive errors like “cannot write to /dev/ulog_main, invalid argument”). We also notice some issues with firmwared not setting up the virtual PTY for init-time logging. As such, boxinit sends all logs to /dev/null, and we have no way to access them. We developed a workaround by strace-ing the PID that would have been writing them and looking for any write() calls to FD #2 (stderr). This allowed us to understand some of the other issues we were facing. However, having easy access to the logs would have made things much easier and saved us a lot of time.

Closing Remarks
While there were some issues getting off the ground (lol), we found Sphinx to be an immensely useful tool, particularly since we didn’t have to crash any of our real drones to figure out how the flight plans worked. It can be a bit difficult to work with, but is definitely worth the learning curve. Here is a flight plan that we used to test image capture working within Sphinx in the world file called “outdoor_1.world”:

QGC WPL 120
0         0         3         178     0.0      3.0      -1.0     0.0      0.0      0.0      0.0      1
1         0         3         93       20.0    0.0      0.0      0.0      0.0      0.0      0.0      1
2         0         3         50001 1.0      10.0    0.0      0.0      0.0      0.0      0.0      1
3         0         3         205     -90.0   0.0      0.0      0.0      0.0      0.0      2.0      1
4         0         3         22       0.0      0.0      0.0      0.0      0.0      0.0      0.0      1
5         0         3         16       1.0      2.0      0.0      0.0      48.87884073471497 2.367869801697479 20.0    1
6         0         3         16       1.0      2.0      0.0      45.0    48.87895889061195 2.367690138640655 20.0    1
7         0         3         2000   0.0      1.0      12.0    0.0      0.0      0.0      0.0      1
8         0         3         16       1.0      2.0      0.0      90.0    48.87895889061195 2.367869801697479 25.0    1
9         0         3         2000   0.0      1.0      12.0    0.0      0.0      0.0      0.0      1
10       0         3         16       1.0      2.0      0.0      180.0  48.87884073471497 2.367690138640655 30.0    1
11       0         3         2000   0.0      1.0      12.0    0.0      0.0      0.0      0.0      1
12       0         3         16       1.0      2.0      0.0      270.0  48.87884073471497 2.367869801697479 35.0    1
13       0         3         2000   0.0      1.0      12.0    0.0      0.0      0.0      0.0      1
14       0         3         16       1.0      2.0      0.0      0.0      48.87884073471497 2.367869801697479 35.0    1
15       0         3         21       0.0      10.0    0.0      0.0      0.0      0.0      0.0      1

To any Parrot devs, we are more than happy to help with writing bug reports or documentation, just let us know how we can best get this information out to the right people!

1 Like

Hello @anash, thanks a lot for this insightful report, we will look into it with R&D and will get some answers really soon!

1 Like

@anash

Perhaps you’ll find these documentation links useful (the first one being most relevant for your use case - I see you already have it too):

https://developer.parrot.com/docs/mavlink-flightplan/
https://developer.parrot.com/docs/refdoc-android/com/parrot/drone/groundsdk/mavlink/package-summary.html

I’m intimately familiar with the first link - more documentation on the commands would have been helpful, although we understand most of them now (with the exception of 20 - return to home command).

As for the second link, I’m not entirely sure if I’m reading the page correctly, but I wasn’t able to glean anything particularly useful from it besides the fact that all of the commands extend the general MavlinkCommand. Is there something I’m missing?

2nd link is a higher level java wrapper for generating the mavlink flies. If you use groundsdk it is built-in to the SDK.

Some other things I recall about the Parrot mavlink implementation from experimentation over the years:

  • Too high of a speed between two waypoints with too low of a loiter value will cause the drone (real world – this isn’t just in sphinx) to stall. This could makes things very tricky if your drone has travelled beyond radio range.

  • Unnecessary / duplicate VIEW_MODE commands can cause the drone to fail loading the mavlink file. There are a couple other scenarios where the drone will simply ignore a mavlnk file, looks like you found one of them (no waypoints).

  • Placing a takeoff command too far away from a prior land location will cause the drone to takeoff but not proceed to the takeoff location, causing the drone to stall. However, the drone has no problem taking off to reach its first waypoint if a take-off command isn’t specified (I think you found this one).

  • The Disco drone requires TAKEOFF and LAND commands to be set otherwise it refuses to execute the mavlink file.

  • MAV_CMD_PANORAMA_CREATE is completely illogical. It relies on RELATIVE angles and is a major PITA to get working properly.

  • MAV_CMD_NAV_RETURN_TO_LAUNCH execute the drone’s built-in RTH routine. Set this command to have the drone RTH using its predefined RTH altitude.

Shell

I just remembered another decent source I used for cracking parrot’s mavlink implementation.

This is Parrot’s older mavlink file generator. Within it you’ll find all the parameters the drones support with some (basic) notation around specific parameters.

An example here:

  • Take a look at ARMAVLINK_MissionItemUtils_CreateMavlinkVideoStartCapture. You’ll find that the MAV_CMD_VIDEO_START_CAPTURE command has provisions (and accepts) specifying a framerate and resolution. These used to be required parameters too.

That’s about all I got. Hope this helps,

Shell

Here’s another:

MAV_CMD_DO_MOUNT_CONTROL

Negative degrees point the gimbal down. Positive ones point it up.

And, it is absolute :slight_smile:

These are some great tips, thanks! I haven’t used the VIEW_MODE or the PANORAMA options, but this will definitely help with future experimentation.

I think I’m a bit confused about the RTH command. If I recall correctly, I’m thinking maybe I had expected it to land (in Sphinx) but it just hovered. Instead of using it, I’ve been setting an explicit waypoint for the drone to come back to where it launched from, followed by a LAND command. Would the RTH supplement the final waypoint but still require an explicit LAND? If so, does it maintain its current altitude? I’m also wondering how the RTH works as far as setting a precise home location within the app

Also a number of these things ( MAV_CMD_DO_MOUNT_CONTROL, MAV_CMD_SET_STILL_CAPTURE_MODE, etc.) I had to figure out with trial and error. It would be nice if these were better documented, and without Sphinx (which was quite the hassle to get up and running with) I would have wrecked a drone for sure. For any Parrot folks out there, more than happy to help with documentation!

The timing of your post was perfect as I’ve been doing some final validation with mavlink.

I have some further details given nuances across drones I’ll share once at a PC as I have to test B1, B2, Disco, and Anafi.

You got RTH correct. When activated it will 1.) rise / fall to the set RTH altitude 2.) travel directly to its designated home location and 3.) descend to a couple meters altitude.

The home location is based on what you have set, or what the drone has chosen, as the desired home type - takeoff, controller, tracked target, first fix location.

For whatever reason, the Anafi. requires View changes to occur AFTER the waypoint you want them to be active for while traveling to it.

Notice this snippet:

1	0	3	16	    0.000000	10.000000	0.000000	282.658279	40.205570	-75.639126	10.000000	1
2	0	3	50000	1.000000	-1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1

You would think that the view type would not change until AFTER the waypoint is reached, but in actuality, the Anafi drone executes the view command immediately after initiating travel to the waypoint.

This is the exact opposite behavior of the Bebop drones. View mode must be set before the waypoint entry for it to be active while traveling to that waypoint on the older drones.

This applies to all 3 View types, RELATIVE, ABSOLUTE, and ROI.

I had to make a video of this one as I think it is a bug.

and its mavlink file:

QGC WPL 120
0	0	3	178	    1.000000	10.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
1	0	3	16	    0.000000	10.000000	0.000000	353.337684	40.205570	-75.639126	10.000000	1
2	0	3	50000	1.000000	-1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
3	0	3	16	    0.000000	10.000000	0.000000	329.379082	40.205770	-75.639281	10.000000	1
4	0	3	16.  	0.000000	10.000000	0.000000	30.793737	40.206116	-75.639011	10.000000	1
5	0	3	16	    0.000000	10.000000	0.000000	116.085812	40.206030	-75.638781	10.000000	1
6	0	3	16  	0.000000	10.000000	0.000000	206.483858	40.205564	-75.639085	10.000000	1

Pretty cool right? The drone “progressively” makes it course through the waypoints without dramatically reducing its speed or outright stopping between them. Loiter distance comes in handy here as it smooths the turn between waypoints too.

Here’s what happens when you insert a MAV_CMD_DO_MOUNT_CONTROL command:

QGC WPL 120
0	0	3	178 	1.000000	10.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
1	0	3	16  	0.000000	10.000000	0.000000	282.658279	40.205570	-75.639126	10.000000	1
2	0	3	50000	1.000000	-1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1
3	0	3	16  	0.000000	10.000000	0.000000	329.379082	40.205770	-75.639281	10.000000	1
4	0	3	205 	-10.000000	0.000000	0.000000	0.000000	0.000000	0.000000	2.000000	1
5	0	3	16  	0.000000	10.000000	0.000000	30.793737	40.206116	-75.639011	10.000000	1
6	0	3	16  	0.000000	10.000000	0.000000	116.085812	40.206030	-75.638781	10.000000	1
7	0	3	16  	0.000000	10.000000	0.000000	206.483858	40.205564	-75.639085	10.000000	1

Note the drone come to a complete stop at waypoint #2? Further on in the video I toggle to a flight view to verify the orientation was adjusted -10 degrees (you can see the white dot in the HUD is slightly below center).

Time management and image capture
According to the description of the problems you met and the way you workarounded them, everything seems to be related to the simulated clock domain. As Sphinx may or may not be able to achieve a full real-time factor (100%) according to the available GPU/CPU resources on the PC, simulation engine and simulated firmware cannot use their own clock. So the firmware is running inside the simulated clock domain, which is totally different from the host system’s or from the smartphone running the controller.

The simulated clock starts at zero by default. But as you discovered, it is possible to set the start time to any arbitrary value. However, this option does not solve everything: the simulated clock is monitonic but can be slow and irregular, depending for example to what is running in parallel on the host. So there is - almost - always an unpredictable drift between the system clock and the simulated one.

For all these reasons, you may have problems if you assume that your application and the drone are synchronized on the same clock.

Image resolution with anafi-pc
Image resolution is limited to 1280x720 for performance reason. It is not possible to increase the resolution in a simulated environment.

logs
The public version of sphinx does not offer access to firmware logs.
Sphinx itself (not the simulated firmware) does not use Parrot/Android log system (i.e. ulog), it relies on gazebo log system, which posts directly in the shell holding the sphinx process.

As promised, here is the QGC flight plan conversion code! Sorry for the delay, wanted to get everything cleaned up nicely: https://github.com/slingshot-aerospace-inc/parrot-qgc-flightplan

It’s far from perfect so if anyone has any suggestions/wants to help, please don’t hesitate to do so!

1 Like