jdl

  • ***
  • 246
Yesterday I experienced a really strange behaviour from my Revo Board on a ZMR250 with M8N GPS & I2C MAG, OPLINK control & data, flying in RATE mode (LP "next" r957). I have to add that this flight was with higher AUW than usual (5200mAh Lipo & 50gr. HD camera).  But I've flown numerous time with this setup using r957 and previous "next" build versions without problems. Also with lighter batteries: 2200mAh & 1800mAh.

The purpose of the flight was to test different QuickSmoothValue setups after AutoTune to achieve a balance between best handling and smoothest video shooting possible with this heavy frame.

The first glitch occured during a moderate accent, maybe 30 meters above ground. There was a brief slight roll (10-20 degrees maybe) to right, without my command. After the jerk the flight continued normally but I got worried. Made some circles around at this altitude and decided to land. The QuickSmoothValue of 0.5 appeared to be too high and inappropriate for a smooth video shoting as I felt the copter little "nervous" in the wind (8-12km/h, just a light breeze). So I descended cautiously and when the copter was maybe a meter over ground, just a second after I raised the throttle (definitely not full throttle!) to stop the descent, then the copter rolled to the right on its own at about 120-130 degrees, stayed (stopped rolling and stabilized itself) in this orientation for a fraction of a second and then went to the ground. I've managed to cut throttle fast enough so the only damage was the GPS stand, not a big deal.

The crash was definitely neither a pilot error, nor a mechanical or electrical failure too. And I'm almost sure it was not a desync issue too. And the motors and ESCs were cool. Motors are Emax MT2204/2300KV, ESCs - LitteBee 20A.

Later I examined the telemetry record from the GCS and the video from the HD camera. Link to the video of the crash is attached.

Playback of the telemetry record shows that when these sudden rolls occur, they are preceded (or happen at the same time) by alarms in SystemHealth Display. First, ATTI goes orange and STAB goes black with red X.
Next ATTI goes green, STAB goes orange. After a while the STAB goes green too. There is also a CPU Overload alarm recorded
immediately after ATTI recovers. It seems that for some reason the Revo loses momentarily the ability to stabilize the frame. No other alarms (including NoGPSFix) are recorded at this moment (checked this in a Scopes window showing all possible alarms).
The raw data flow from gyro sensors seems to be uninterruped, as far as I can understand the telemetry record.

I attach screenshots of the telemetry playback at the moment of the crash, also the telemetry record from the GCS itself. The alarms are present at 248-250 and at 401-403 seconds since start.

I have no ideas where to look for the problem resolution. I can replace the Revo board with a spare one I have but this doesn't seem to me to be a hardware problem. I can also try the latest RC2 hoping that if this was a bug in the Revo firmware r957, it will not persist in the latest builds. But as the problem is not easily reproducible, I would not know with confidence that the same crash won't happen again, this time from hundred meters above ground or over some water areas.

Crash video:

GCS snapshots and telemetry log are in attachments.

I hope that someone would be able to analyse the log file and give a reasonable explanation of what happened.


P.S. For some reason youtube link is considered "unsecure" and not displayed by default. Here is the link itself, just remove dashes between "https" letters.

h-t-t-p-s://youtu.be/ZN4sSoIA-vM
« Last Edit: October 10, 2016, 10:44:25 pm by hwh »

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #1 on: October 12, 2016, 03:58:33 pm »
Thanks, hwh,

Now I see what was wrong with the youtube link.

I'll upload the UAV file that corresponds to the Revo configuration used for that flight. Hope it can bring some clarification.

Meanwhile updated to RC1 but weather is bad and not suitable for testing. Did a short flight indoors with RC1 in Complementary Mode, no problems. This Revo board has many, many hours flight time without issues till now...

Still suspect bug somewhere in the firmware because of the STAB and ATTI and CPUOverload alarms but I'll be happy if someone can prove that this is a hardware problem with the FC. Can a bad gyro cause these alarms (not sure!)? No other alarms and the data flow from the gyros seems uninterruped in the telemetry log.

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #2 on: October 17, 2016, 12:07:26 pm »
Unfortunately migrating to 16.09-RC1 helped nothing. A crash happened again, even more voilent. This time I test flew with 1300mAh 4S and 5045BN props. PIDs were set with Autotune, then QuickSmooth moved to 1. The same setup and similar PIDs I've used with success for months on my other ZMR250 frame that is almost a twin one to this, except flight controller (classic CC3D) and 2.4GHz receiver. I use the other (non-autonomous) frame primarily for practicing flying in the woods :) so I need these higher PIDs for more precise control.

I attach the telemetry log and also videos from OSD and HD camera in normal and slow motion.

Telemetry logs, although useful, obviously can lack some critical data if there are missed OPLINK RX packets or the FC was too busy (overloaded) at the moment.

Again telemetry playback shows ATTI & STAB alarms and CPU Overload in the moment of the flip/crash. Maybe a coincidence, but when the quad went crazy, it made a flip at some 550 degrees (judging from HD video playback), the same as MAX RATE setting for this bank.

OPLINK telemetry is missing this, but on OSD it's visible that just before the flip there was a NO GPS FIX situation. The best guess I have is that in the latest LP builds in some circumstances EKF goes nuts if large deviations in data from sensors are presend for longer time (as oscillations / jerkness captured by gyros/accelerometers or GPS or MAG variations) and it loses realistic attitide estimation. That should not lead to failure to stabilize the frame in RATE mode using only gyros. But that's just the case - STAB Failure happens and the frame rotates uncontrollable. Funny - I can see on the slow motion video that after the 550 degrees flip the FC stabilized the frame, of course it was upside down. Although being disorientated, I understood that I'm inverted and tried to roll the frame. But either my reaction with the stick wasn't determined enough or the FC went crazy again after a fraction of a second (there are two consecutive ATTI/STAB alarms recorded in .opl file) and the quad crashed.

I have a new spare Revo controller and can replace the current one but I don't think it's a hardware sensors/controller failure. I'll revert back to some of the non-autotune LP "next" builds (r498 or r616) that didn't show such problems for me. I'll report back the results of the tests, of course.

Meanwhile I hope someone from the developers to be able to look at the records and to express an opinion on the  issue.

HD Cam:


HD Cam Slow Motion:


OSD:


OSD Slow Motion:


f5soh

  • *****
  • 4572
    • LibrePilot
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #3 on: October 17, 2016, 01:07:00 pm »
Looking at your log there is something strange...

StabilizationDesired.Roll and RateDesired.Roll go to max while the flip is on Pitch axis only.
Same for AttitudeState that report a Roll motion while in your video that clearly Pitch

« Last Edit: October 17, 2016, 01:25:01 pm by f5soh »

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #4 on: October 17, 2016, 01:26:54 pm »
@f5soh: I think this in not the moment of the crash. I did one tes roll soon after takeoff, I suppose you are looking at it in the telemetry snapshot you provided. The crash occurs at 80-82 seconds after takeoff.

But I saw another strange thing: telemetry says actuators on channels 2 & 3 went up almost to the max during the flip, but this should cause flip in the opposite direction, not as captured on the video... Oherwise, the current draw during the flip roughly confirms that two motors are almost full throttle while the other two are idle.
« Last Edit: October 17, 2016, 01:42:37 pm by jdl »

f5soh

  • *****
  • 4572
    • LibrePilot
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #5 on: October 17, 2016, 01:38:58 pm »
Motor 2 and 3 at max should give a Roll flip...

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #6 on: October 17, 2016, 01:40:40 pm »
I see that in the scope counting the motors starts from 0. Number 0,1 - front motors, 2,3 - rear motors. I mean ActuatorCommand.Channel0 stays for NW motor, Channel1 for NE, Channel2 for SE, Channel3 for SW one.

P.S.

This is the StabilizationDesired and RateDesired Scopes snapshot of the real crash moment:
« Last Edit: October 17, 2016, 01:56:49 pm by jdl »

f5soh

  • *****
  • 4572
    • LibrePilot
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #7 on: October 17, 2016, 02:03:11 pm »
You are right for motors, and looked at wrong flip... not the last.


jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #9 on: October 17, 2016, 02:41:02 pm »
@f5soh: Thanks a lot for the help, and for pointing the code section!

I now understand the cause for the STAB failures and consequent crashes. I had some vague doubts about missing data from gyros but not knowing where to look in the code and not having enough expirience in that field I chose to suspect a firmware bug...

Now I'll replace the problematic Revo FC with the spare one I have. Can you give any clue based on your expirience what's wrong with gyros? Cracked solder joint maybe? The problem is intermittent. Do you think I have a chance to repair that FC board? Just for the sport :)
I recently successfully replaced and revived the STM32F103 of one CC3D board that I've burnt by accident. Resoldering MPU6000 wouldn't be harder, maybe...
« Last Edit: October 17, 2016, 02:56:09 pm by jdl »

f5soh

  • *****
  • 4572
    • LibrePilot
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #10 on: October 17, 2016, 03:02:12 pm »
Reflow will help for fixing bad solder join in this case.

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #11 on: October 17, 2016, 03:31:35 pm »
I'll try.

One more question: when may SENSORS alarm be triggered? Only on boot/init? Shouldn't missing communication with MPU6000 due to bad solder joint raise more alarms, different from ATTI/STAB? If so, then in my case the communication with MPU6000 over SPI should be ok, that leads to me thinking of internal damage of the gyros module inside MPU6000. Resoldering will not help in this case, only replacing the IC.

f5soh

  • *****
  • 4572
    • LibrePilot
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #12 on: October 17, 2016, 03:44:43 pm »
Sensor alarm is triggered in sensor.c

No idea about internal damage or not

jdl

  • ***
  • 246
Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #13 on: October 17, 2016, 04:07:43 pm »
Thanks again for your time! I appreciate it!

I'll look in the code.

Re: Strange ATTI & STAB Alarms during flight in RATE mode -> roll & crash
« Reply #14 on: October 19, 2016, 12:02:14 am »
I have had similar issues with two different causes:

When running INS13, even when not using a GPS flight mode, the stabilization is affected by "GPS jumps".  I have a log where I was flying Attitude mode and it banked over about 30 degrees and flew into a tree.  This was caused by a GPS jump throwing the attitude estimation off.

Marginal I2C mags cause all kinds of problems, all the way to rebooting in mid flight.  There are some aux mag GPS's that I won't fly because they are too marginal.  Shorten your I2C wires and twist them with some ground wires?  Use the latest 16.09 RC.  16.09 RC#something has a fix to add some pullups to make these signals more robust.

Before you do that, find a 12v power supply and attach your quad to that instead of a flight battery.  That will let you run it for a long time.  Now power it up, plug it into USB on your computer and note the time you power it up.  Later, go back and check the timer in the Firmware tab.  It shows seconds since power up.  Calculate what it should show given the time you powered it up and the time it is now.  Like after two hours it should show 7200.  If it shows less, it has rebooted while you weren't watching it.  This rebooting is usually caused by bad I2C aux mag.