Wind from one or more props is a major concern too (not just light). Even after trying to keep the wind off, on one quad I have, it would climb at full power if I yawed left hard (and not stop if I neutralized the yaw), but not if I yawed right.
You need just the right thickness of "open cell" foam. The right thickness is like the right sized tiny hole, and it slows down the changes from prop blast, but not from actual altitude change.
Both VelocityRoam flight mode and AltitudeVario thrust mode have a dead band at center throttle stick. If the stick is within +-10% (IIRC) of the center it acts exactly like it is in the center and is not supposed to vary in altitude. Some users report they have to tune the Altitude PIDs to get it to work better. I don't really recommend that.
Loosing altitude if going past half pitch stick when in these modes is usually a sign of motors not strong enough to do this. As you approach a 90 degree bank the power required to avoid loosing altitude goes to infinity so all quads fail sooner or later.
Did you start from a cloud config or someone else's config (UAV file)? That sometimes causes confusion where the original quad was tweaked in some strange way...