Hi, nacholibre
For flight controller to recognize where North and South is, you need working and calibrated Magnetometer.
For normal flying Magnetometer is not needed, it is only needed when flying in autonomous GPS flight modes. Complementary+Mag is not really needed for normal flying.
You should set to Complementary and it should be fine. If you Magnetometer has interference it will only make things worst.
If your board drifts a little it is ok and normal. If it drifts too much, then it could be for many reasons, bad frame, uneven mass, vibrations, bad build in general or bad calibration.
When I fly in Complementary only and I tune my quad-copter, I can use Altitude-Hold flight mode (with tuned altitude PIDs). Then it can hover such that I can put Taranis on ground for 1-2 seconds, pick it up and it does not drift too much (was no wind when I did that). I don't even use GPS for that.
If your build drifts that you can't even let the sticks, then it's probably something bend, lose or bad calibration. This is not really an issue for FPV or racing I think, but if you want at some point step up to GPS flight modes, then you have to fix that issue first and make it fly good in Complementary.
P.S I would start with checking calibration, if it was made on level surface etc..