Quad will not arm in EKF unless Home is set.
Also Home and Base are two different things used for different purposes.
Home is used for GPS coordinates. When using EKF, the coordinates are relative to Home.
Base is just where it returns to when you do RTB. The default for Base is that it is set each time you arm a new battery (once GPS gets a good fix).
Home should be set "close to where you fly" but doesn't generally hurt anything if not. Example: If my Home is set to flying field (50km away) and I fly EKF at my house (50km away from where Home is set), my coordinates on the GCS map are off by several hundred meters, but it is all relative so it just doesn't matter.
It doesn't fly away if Home is a long way away, it just means that it thinks it is here when it is there. For me it was 300m wrong when Home was 50k away. It really only hurts if you are using waypoint flight, like trying to fly between trees on the map. The map will be wrong (by e.g. 300m) and thus the trees will be hard to avoid.
PH flyaway (really this exact thing happens in any GPS mode) would usually be caused by bad mags or by GPS failure after takeoff. One time it happened to me and I had a log, I researched it and found that the mags were bad at high power in the air. Another time it happened to me because I couldn't get it to arm with my FPV switched on, so I switched FPV off, armed, then switched FPV back on, then flew (with a GPS signal that had gone bad). I learned...
Your high current (motor) wiring MUST be twisted correctly and it is a good idea to test the mags at high motor power. Make a log in a high power test flight that does not use EKF or GPS and make sure the mags stay green no matter what direction you fly. Better yet, strap it down to your back porch, and watch the mag scope while you run the motors up to high power. It should change very little (like 10% or less at full power). This must be done with props on so there is a lot of current flow. Be careful.
One way that might make it easier to "hold down" is to swap props so it pushes down instead of trying to lift off. Take all props off and put them on (upside down if you like) the motor to the immediate CW (or CCW).
I wrote this quickly. I hope it helps.