[TheOtherCliff]

Update: 2020-11-27 Relationship to telemetry(/OSD?) Disproven?

The relation to telemetry was proven false in further testing.  That implies that the relation to OSD (a form of telemetry) also needs to be retested.

I can recreate this issue (~ 4Hz oscillation in VelocityRoam mode) every time (but only when vehicle is pointed south).  I tried both leaving GCS telemetry powered off and completely disabling the quad FC's OpLink and the oscillation still happened.

In relation to oscillations that occur when using OSD-telemetry (https://forum.librepilot.org/index.php?topic=4923.0), I have basically confirmed that it happens with normal telemetry too.  It happens with my 16.09 Sparky2 DJI GPS quad when using telemetry in VelocityRoam.  Not in Attitude mode.  Further testing is needed.

For VelocityRoam (and I presume any similar GPS mode) some setups have "steps" or fast oscillations in stabilization.  Instead of quickly becoming level it takes small steps toward level about every 250ms (time is guessed).  Strangely, stick response is still immediate, so it acts like it's concept of level is oscillating, but not the stick control response that is relative to that concept.

Years ago I saw the GCS attitude oscillate when the quad was motionless on the ground.  Recalibrating the mags seemed to fix that.  I want to get all the info I can get before I recal the mags and the problem seems to go away again.  The new/old WorldMagModel may even have an issue here.  Some of my quads have the new model and some the old.  Some of my quads have my DJI GPS oscillation fix (as posted on the forum).  Not sure about this quad.

It appears that several similar issues may all have the same cause.  This has been seen even before 16.09 and with Ublox GPS.  Both the pre-16.09/Ublox-GPS and my 16.09/DJI-GPS have a fast east/west oscillation that can become this fast-stepped-recovery to a slow 4 second oscillation for my 16.09/DJI-GPS and .  This quad is tuned fairly tightly with AutoTune with a high max roll rate, so the attitude commands from VelocityRoam have a very quick response.  It may be that loosely tuned models would not show the stepping, and only the slow oscillation.

One possible reason that also needs to be explored is that pirouettes (even very slow ones) with this quad, cause a lateral drift that is related to compass heading.  When you do a full 360, the drift comes back to the original location.  This implies that the GPS antenna has a directional offset?  One of the things on my list :slightly_smiling_face: is to add a GPS mount offset so that this pirouette is stationary.  This issue raises questions about bank angle though.  Does the GPS have a bank angle offset too?  That would cause this issue too.  I need to rotate the GPS 90 degrees, change aux mag orientation, and see if the oscillation is still east-west or whether it is now north-south.  This seems to be caused by bad sensor calibrations (accel and mag) so it's hover position is in the direction of non-level drift with the GPS constantly trying to pull it back.  You yaw rotate and the drift is in a different direction but it tries to bring it back to the "Position Hold" location.

Then there are all those unneeded telemetry packets that I found some years ago; remove them all to see if it is congestion related.  Maybe try really reducing all the telemetry for a test.  It is odd that increasing a telemetry packet rate makes the fast oscillation / steps quicker (in time with telemetry) and thus less of a problem.  It would seem that reducing the telemetry packet rate would slow the fast oscillation down and make it worse.  Slowing it down should reduce congestion if that is an issue.

Further testing is needed ... to recreate the issue, and to see that it goes away if telemetry is disconnected or disabled entirely.  To see if the new WorldMagModel helps.  To compare with/without the DJI oscillation fix code that I posted on the forum some time ago (recall though that it happens with Ublox too).  To see if reducing the AutoTune PIDs is warranted.  To see if a coded GPS mount offset (pirouette drift out and back) is warranted.  To see if GPS bank angle affects the GPS location that it gives to us.  Note to self:  I need to rotate the GPS 90 degrees, change aux mag orientation, and see if the oscillation is still east-west or whether it is now north-south.

I will try to get some logs, examine them for clues, and make videos.

Here is @Trust's recent thread with excellent video of the same problem (watch the video in the first post from 0:18 to 0:30).  @Trust complained of the quick oscillation, but you can see that there is the same slow oscillation too.
https://forum.librepilot.org/index.php?topic=4923.0

Here is a video from years ago.  The first 10 seconds show the large slow oscillation with steps.  The rest of the video is full of recording dropouts and is worthless.
password is oops


Edit 20201125: Here are some possible reasons and factors I can think of for these oscillations.  If you think of one not mentioned, feel free to post about it here:
- See if it goes away if telemetry is just disconnected (coordinator powered off) or if telemetry must be disabled in the model or if just drastically reducing the number/size of packets makes it go away.
- Simple priority issue.  Both telemetry and stabilization must be high priority.  Loosen the tight 2ms telemetry reply window and reduce the telemetry priority.  What does this do to OpLink RC control?
- Examine captured telemetry: See if some of the embedded debug shows signs of a problem:
   ActuatorCommand->UpdateTime, MaxUpdateTime, NumFailedUpdates
   ActuatorDesired->UpdateTime, NumLongUpdates
   CallBackInfo
   EKFStateVariance
   SystemAlarms
   TaskInfo
   etc
- Does increasing VtolPathFollower->UpdatePeriod (to more than the telemetry period) have an affect?
- Add debug to watch the starving of stab by telemetry "after the fact" since telemetry is part of the problem
- Try to time the period exactly.  That may hint at what it is related to if it is a nice number like 250ms.
- Change the PID tunings and/or vehicle weight to see if the oscillation period changes.  If it changes, the problem is not related to a timed process.

The following aren't fixes since it only occurs with telemetry enabled.  They are probably only interesting side projects at this point:
Interesting related topics:
- See if the new WorldMagModel helps.
- Compare with/without the DJI GPS oscillation fix code that I posted on the forum some time ago (recall though that it happens with Ublox too).  DJI GPS fix code helps the slow oscillation, but not the fast oscillation.
- See if reducing the AutoTune PIDs is warranted.  Probably not since user @trust has the issue without using AutoTune to tune PIDs.
- Rotate the GPS mounting say 90 degrees and correct for this in aux mag orientation settings.  See if the problem is still an east west oscillation when facing south (north with some vehicles).
- See if a coded GPS mount offset (pirouette drift out and back) is warranted.
- See if GPS bank angle affects the GPS location that it gives to us.
- Imagine what an incorrect WorldMagModel value would do.  If magnetic inclination is wrong then there would be 0 or 2 headings that would match.  Imagine a cone pointing straight down from the vehicle.  The magnetic field should be a line on that cone.  If the mag vector falls inside the cone then there are two roll angles that put the vector on the cone.
- Hand code a WMM inclination offset and vary that to see if the oscillation changes.
- Try old EKF settings.
- Oscillation seemed significantly worse at the PCMA field than at the house 35 miles away.  Both using the same HomeLocation (at PCMA).

[TheOtherCliff]

It appears that the fast oscillations are caused by bad sensor calibrations (mag and accels) telling it that level is like this (which is wrong), but GPS telling it that it should come back when it drifts off.  The result is that it stays half way between sensor level and GPS correct position, getting tugged one way and then the other.  I make this educated guess because it fits the facts and then was backed up by the fact that halving the GPS packet rate halved the oscillation frequency.  That is, it wasn't a coincidence that the GPS packet rate was exactly the same as the oscillation frequency.

I think that one real test of whether you have bad sensor calibrations is whether your VTOL drifts significantly away when you do a very slow pirouette.  My "bad" test drone drifts several meters if I simply yaw it 180 degrees and becomes quite unstable if I try to do pirouettes that are anything faster than very slow.  Again, I haven't tried to recalibrate this bad drone till I understand the problem more.

I don't know whether the bad sensor calibrations of the fast oscillation could affect the slow oscillation.  I can tell you that with a bad mag calibration and the model sitting motionless on the ground for 5-15 minutes, I have seen the GCS PFD display start (an at least similar) slow oscillation, and ATTI or STAB go from green to yellow or red I forget which.

My slow oscillation recreations are finicky.  Some days they are bad and some days I only get fast oscillation.

Long ago I tried changing VtolPathFollower PIDs and didn't get rid of the slow oscillation.  I have assumed that there is a tuning piece that I just couldn't find.  I found some possible tuning pieces but have not tested at all.  The tuning that I found is related to the statements in revosettings.xml so RevoSettings in System -> Settings.  Tweaking these could help:
        <!-- Low pass filter configuration to calculate offset of barometric altitude sensor.
        Defaults: updates at 5 Hz, tau = 300s settle time, exp(-(1/f)/tau) ~= 0.9993335555062
        Set BaroGPSOffsetCorrectionAlpha = 1.0 to completely disable baro offset updates. -->
        <field name="BaroGPSOffsetCorrectionAlpha" units="" type="float" elements="1" defaultvalue="0.9993335555062"/>

   <!-- Coefficient for velocity estimate post processing low pass filter
        - filters velocity bias based on delta position to compensate offsets coming from EKF -->
   <field name="VelocityPostProcessingLowPassAlpha" units="" type="float" elements="1" defaultvalue="0.999"/>

There is a related piece that may not be part of the cause, but i would like to mention it here.  @JDL found that VelocityRoam mode does not have any deadband.  Most transmitters do not output the exact same position from one stick release to another, so it was not changing into PositionHold.  I think that the worst this meant was that VR mode would drift in the wind if you let sticks centered.  This uncovers another issue in that telling VR to fly slowly north when there is a wind from the west would actually have it flying north-east.  I haven't seen him post about this in the forum.  It is a trivial change if you build your own firmware already.
https://librepilot.atlassian.net/browse/LP-620

[jdl]

...
It is a trivial change if you build your own firmware already.
https://librepilot.atlassian.net/browse/LP-620

Regarding LP-620, there is probably a cleaner workaround without need to build new firmware.

Looks like that just setting (in current "next" firmware, probably also in LP16.09) the Assisted Control to GPSAssist for VelocityRoam mode in GCS: FlightModeSwitchSettings tab  enables the persisting AssistedControlStickDeadband (default: 8%) in RemoteControlInput tab for VelocityRoam and this would solve the issue with VelocityRoam drift in winds.


Edit: Replaced the image with deadbands, it could mislead to set the Roll/Pitch/Yaw stick deadband to 0, this is not recommended and was only for testing purposes. Leave it to default "2" or raise a little if necessary (depends on your transmitter).

The other deadband, Assisted Control stick deadband is only active for GPSAssist modes. However, if it is smaller than Roll/Pitch/Yaw stick deadband, the larger value from both is used in GPSAssist modes.

[trust]

In the last few test flights on a quad, In VR mode I've noticed if there was any wind more than 5mph, the quad would drift with the wind, yet it is easily compensated if I manually fly it. I'm using a Futaba 9, and it seems to be a very repeatable value when looking at the inputs, so I'm not sure 8% is really needed, but since it only operates in VR mode, probably ok to try the GPSassist with deadband.

[TheOtherCliff]

My recollection is that you can't use GPS-Assist with any GPS flight modes... at least in 16.09.  GPS-Assist is to e.g. add a PH to Attitude mode if you release the sticks.

----------------------------

The quad doesn't drift with the wind in true PH mode.

There are two problems there.  The fact that it drifts with the wind means that is not actually in PH mode.  It should be in PH if the sticks are centered in VR mode.  That is what the deadband fix is for.

The other problem is also that it drifts with the wind.  Currently, if it is pointed north and I give it forward it flies north.  The problem is that if there is a wind out of the west, it also drifts east a bit, kind of like Attitude mode in the wind.  It doesn't have to be that way.  An enhancement would have it flying straight north, even in wind.

Pointed north, with a wind from the north and a very small forward stick, it will actually fly south.  We already see this by the fact that it drifts with the wind with a very small amount of stick to keep it out of PH.

[jdl]

My recollection is that you can't use GPS-Assist with any GPS flight modes... at least in 16.09.  GPS-Assist is to e.g. add a PH to Attitude mode if you release the sticks.

I've thought the same. However, config tool (in "next") allows to setup GPSAssist for VelocityRoam and Save settings without error/warning message. It seems logical as VelocityRoam IS GPSAssist-ed when sticks are released (it enters Brake, then Hold).

Setting GPSAssist for VelocityRoam just sidesteps a (my suggestion) missing logic in
 …\flight\modules\ManualControl\manualcontrol.c

Code: [Select]

static uint8_t isAssistedFlightMode(uint8_t position, uint8_t flightMode, FlightModeSettingsData *modeSettings); that needs correction to always recognize VelocityRoam as assisted flight mode, so the dedicated AssistedControl sticks deadband is applied for VR. I've commented this in LP-620 https://librepilot.atlassian.net/browse/LP-620

I plan to test this in real flight tomorrow...

P.S. I'd also try to set GPSAssist for AutoLand flight mode (in config tool). This should allow position corrections during descent (at least the pop-up hint for the AssistedControl field says so...)

[jdl]

I've just returned from my test field

CONFIRMED!

Setting GPSAssist in Assisted Control for VelocityRoam flight mode in GCS: FlightModeSwitchSettings tab enables applying the AssistedControl sticks deadband (default 8%) and thus the VelocityRoam does not suffer from drift in winds (enters PH when sticks are centered)!

This might be used as an easy workaround for the VR drift issue until the firmware is fixed. No side effects noticed (not that I'd expected any).




Another nice surprise:

Setting GPSAssist in Assisted Control for Land flight mode in GCS: FlightModeSwitchSettings tab allows the pilot to make corrections with sticks during descent, and find the optimal landing spot or just correct position wanderings due to GPS drift.

I've made tests with "next 735" firmware, cannot say if forementioned setting are allowed in LP 16.09 but I suppose so.

[trust]

Ok. Just tested the GPSassist mode in 16.09 on VR mode on my sk450 quad and left the default 8% deadband. Lite west winds - so hard to tell if it was helping the drift issue, but it did seem to be better. In regular STB1 mode, I could hold into the wind with about a -3 degree pitch. Several times while facing west, the craft began the slow east-west oscillation - but this time it kept increasing in amplitude, growing to over ten degrees before I stopped it. If I yawed even 45 degrees - or especially 90 degrees - it stopped oscillating and was quite steady. I did try a little moving around in VR mode and it seemed stable and predictable.

[trust]

Tried this again in somewhat stronger winds (5-10) from the north/nw with some thermal turbulence.
This time the east-west oscillation was there no matter which way I oriented, although more severe if pointed west. It did try to stay in place in the wind - that's good. Moving around in VR mode tended to reduce the oscillation.
  I then tried increasing the gyro settings on pitch and roll - from .003 to .006 (and doubling the ID values also).
This didn't affect the normal flight much at all, but did improve the stability when in VR mode. The oscillation was less severe, but still there.
  I haven't tried autotune but will see what it comes up with.

[trust]

Ran autotune. It came up with PID value in between my settings on pitch - .00423 P and lower in roll - .00240. Yaw was .01059 - didn't know it could go that high! I thought 0.01 was the limit. ID values were scaled appropriately - roughly I=P*2, D=P*0.02125
Yaw D was slightly less. These make sense in that the camera is in front, counterbalanced by the battery behind, higher mass in pitch, so needs more stabilization.
  Manually flown it seemed less "tight" in hover but acceptable. When I flew up and put in VR mode, the same problem appeared - a slow east-west oscillation that increased in amplitude until I stopped it (10s or so).
  It seemed worse than with the higher PID values. At least with the high PID values, it did not get out of control.

[trust]

Also, attitude mode outer loop PI values changed slightly with autotune. My original values were P=2.5 and I=0, autotune gave P=3.514, I=0.824

[TheOtherCliff]

I assume that you are talking about the slow oscillation.  That is the one that sometimes can get worse and worse so you must finally switch to a different flight mode.

If you are using DJI GPS, there is an issue that happens because the DJI GPS smooths the data (and thus delays it) more than the Ublox GPS.  This is the same thing as happens with ESCs using 50Hz PWM... you get oscillation that goes away when you change to 490Hz (or reduce PIDs, but that makes it mushy).

I didn't find a setting equivalent to reducing the PIDs so I wrote a firmware change that tries to unsmooth and speed up the GPS data for DJI.  It generally helps a lot but sometimes not perfectly.  Are you running that DJI mod firmware?

Later on, I found some RevoSettings settings that might help, but I have not tested them.  I described them a while back.

[trust]

Yes, I'm referring to the slow oscillation. Yes, this unit uses the 16.09-dirty firmware with the DJI fix.
Are you referring to these possible fixes?

Long ago I tried changing VtolPathFollower PIDs and didn't get rid of the slow oscillation.  I have assumed that there is a tuning piece that I just couldn't find.  I found some possible tuning pieces but have not tested at all.  The tuning that I found is related to the statements in revosettings.xml so RevoSettings in System -> Settings.  Tweaking these could help:
        <!-- Low pass filter configuration to calculate offset of barometric altitude sensor.
        Defaults: updates at 5 Hz, tau = 300s settle time, exp(-(1/f)/tau) ~= 0.9993335555062
        Set BaroGPSOffsetCorrectionAlpha = 1.0 to completely disable baro offset updates. -->
        <field name="BaroGPSOffsetCorrectionAlpha" units="" type="float" elements="1" defaultvalue="0.9993335555062"/>

   <!-- Coefficient for velocity estimate post processing low pass filter
        - filters velocity bias based on delta position to compensate offsets coming from EKF -->
   <field name="VelocityPostProcessingLowPassAlpha" units="" type="float" elements="1" defaultvalue="0.999"/>

Both of these are at the default values. How could they be changed and what would the possible effect be?
I see you suggest setting the Baro value to 1 would disable it's effect.
What about the other?

[trust]

I'm looking for alternate GPS/mag units that might have better update rates. How about this?
https://www.ebay.com/itm/UBLOX-NEO-M8N-GPS-Module-Compass-Antenna-Engine-For-Pixhawk4-Flight-Controller/303745218271

[TheOtherCliff]

Yes, those are the settings I was going to experiment with.  The comments in the one tell how for that one.  I think that 5Hz is the GPS update rate.  We are working with a slower GPS, so I would try plugging say 3Hz into that formula to see if it works better.

BaroGPSOffsetCorrectionAlpha = 0.998889506 or other number moving farther in that direction if that helps
and the other number, just experiment with say 0.990 0.900 or 0.9999?

I would initially try this on a known platform like a quad as opposed to an experimental platform.

As I recall, Ublox GPSs are all configurable to higher rates.  Maybe some higher than others but M8N's are all the same in that regard.  Change the update rate in GpsSettings, but Ublox GPSs don't have the DJI issue unless you misconfigure them (update rate e.g. 5Hz and smoothing e.g. airborne 4G).  The default is that they are get configured correctly at each power on.  DJI GPSs on the other hand are not configurable.  We are stuck with what we have.