So you say hey both (GPS and FC) use the same compass. Is this a magnetic compass that senses where magnetic north is so I dont need to teach it each time I fly where North is?
If not can you recommend me a compass that does this(sense magnetic north) that will work with Librepilot?
To my knowledge all compass modules sense North. Calibrating magnetometer does not mean it is used, it is the sensor fusion algorithm that you need to set, some use Mag, some dont.
My impression is that many beginners get into thinking that using GPS will magically make aircraft hovering in place of centimeter accuracy.
I guess this thinking comes from the fact that many people have smart-phones and there position from GPS together with sensors, cellular services and wifi gives reasonably good estimates.
This assumption is very wrong and different from model air-crafts (unless expensive ~$1000 RTK system is used). Raw unprocessed GPS alone gives you 3-4 meters of error or worst with cheap antennas. What is very important is that so called GPS flight modes, just do NOT use GPS alone. When you want to use GPS for stabilization with INS31 (EKF) then it will have to use ALL sensors: accel, gyro, magn, baro, gps to work together.
It is the EKF that blends together ALL sensors to give reasonably good position. Despite EKF is designed to deal with noise, it has some tolerance for that. If one sensors gives completly wrong contradicting data, EKF won't be able to give good results.
Adding GPS won't do any magic if there are other problems, for instance noisy magnetometer as just one example.
You should start with Complementary and tune your stabilization PIDs, then try altitude hold/altitude vario and tune your altitude hold PIDs. At that point you should have air-craft that without GPS and no wind, hovers in place without huge drift (it will drift) but allowing you to let your sticks for a couple of seconds. I am able to put Taranis on the ground for 1-2 seconds before it starts driffting and accelerating in one direction.
At that point, when you have resonably stable air-craft you can think of testing Complementary+Mag and set your earth altitude in GCS, to see how air-craft behaves. Magnetometer should be calibrated with full build, outdoors (important that is not indoors), far from metalic obejects, preferably by holding aircraft in the air and using OPLink. Then you do 5 times save and last time you use to rotate your aircraft in all directions. This should be using external magnetometer. You also need to make sure your external magnetometer is in the same orientation as the one on flight controller, since default orientation is 0,0,0 and different manufacturers place external mag on GPS units differently. Mag/GPS+Mag should be high above aircraft on mast that is not rotating (I use ~14cm). Twisting power wires is also recommended to redirect stronger magnetic field horizontally (direction in which wires are going) and make it weaker vertically.
Since any rotation of the mast invalidates calibration, also changes to aircraft like adding camera makes your Mag calibration not accurate.
Once you have Mag error < 5% and always green (never jumps to orange) even when motors are running, then you can try enabling INS31 which will combine all sensors together.