Did you calibrate accelerometers, level and magnetometer (configuration tab, attitude section)? If calibration is OK and your TX trimming at 0 quad should fly level with pitch/roll stick in neutral. Check also your Input calibration for neutral position setup.
BTW, you mention "...when the drone starts to fly..." - do you mean at take off? If yes, than it is ground effect. you should take off relatively fast to about 3 feet elevation, there there is no ground reflection effect.