Were you hovering in Stabilize or in PositionHold? If in Stabilize, very slow drift is almost inevitable, although you can try to repeat board level calibration. If in PositionHold, unidirectional drift is a bit strange, but sometime GPS error could be up to 10 or even 20 m depending on the environment (tall buildings etc.). Horizontal position is mainly dependent on GPS, direction of position correction is dependent on magnetometer. If this direction is wrong the quad movement will look like water rotation while flushing toilet bowl - "Toilet Bowling". I got once swing-like movements, but there was a combination of wrong mag calibration and vibration.
In Stabilize mode some air flow reflected from some asymmetry in motor arms (ESC position, wires, LED etc) can create sufficient force to slowly move your quad in certain direction.