I would put it back on your yaw test table and get the yaw figured out first. You can clearly see the tail oscillating. The tail should lock solid and be responsive and do what is expected before moving on.
Do you know that your head speed is correct? Too slow head speed makes it mushy although to me the roll and pitch looked acceptable but not perfect at 400.
For someone who can fly without stabilization, I recommend adjusting in this order (at flying head speed) because Rate needs Manual throws to be correct, Axislock adds complexity to a correctly tuned Rate, Atti needs Rate tuned first:
- Manual, Manual, Rate adjusting Yaw only PID on rotating stand (or skittering on smooth ground)
- Manual, Manual, Rate in free hovering flight
- Then Rate, Rate, Rate
- Then Atti, Atti, AxisLock
Is there something that you know of that would make your yaw servo slower? Do you have enough amps available (in your BEC) to drive the servo at full speed against resistance? It looks like you are driving your yaw servo at 50Hz. Is is capable of using a faster signal? Is the servo mount and linkage good and tight (little slop or springiness)?
You could be using say 30 degrees for Attitude mode max bank angle, but you are using the default of 55 (you were using 85). If it is still too sensitive for you, reduce these.
Did you use a cloud config (config built into the setup program) or did you start from default settings? I think you started with defaults (as I hope) because I see that the inner loop PID is set to default in both uav files. I worry a bit because sometimes the canned settings have unexpected changes in them. I would start from default (or KarlA) settings. There are several things in settings that can make the tail oscillate, but I don't see any of them.
Does heli yaw still oscillate if you use Rate on Yaw instead of AxisLock?