If your PIDs are too high, you can experience an invisible high speed oscillation. This drives the throttle toward 50% whether the stick is higher than that or lower than that.
If the motors seem to suddenly (not just a slow I term windup) rev up at very low throttle settings that is an indication that this is your problem.
If you are using AutoTune and have a SmoothQuick slider, adjust it to a lower (smoother) value. Or you can simply manually adjust your PIDs to lower values.