So it came back in RTB and them motors shut off instead of landing. I guess you have FlightModeSettings.ReturnToBaseNextCommand set to "Land".
What you describe can happen if the AltitudeHoldSettings.ThrustLimits or VtolPathFollowerSettings.ThrustLimits are set too high. When the Min value is too high, and the FC has reduced the throttle all the way down to Miin, but it sees that it is still climbing, it can decide to disarm and shut the motors off for safety. I haven't read the computer code for that section in a long time, but it that might happen, and it would be worse if the battery was freshly charged, or a lighter battery than normal. In those cases, it climbs even more than normal.