The quad doesn't know the difference between drifting with the wind on a windy day (it doesn't feel the wind then) and windless day. If you were in a hot air balloon in 60mph / 100kph wind and puffed a cigarette, the smoke would stay with you... Think about it.
The only away around this is to have a Revo class FC with GPS flight modes.