In simple terms the procedure is to record home position then use current position and course made good, which the GPS provides directly, to compute the turn correction to return. Scale this correction appropriately and apply it as an offset to the control surfaces.
The Wiki in the gitHub link below gives more explanation.
https://github.com/gke/OpenPilot-RTHOne of my friends had bought a CC3D and found that RTB had been dropped and said that it was stated as being
impossible to restore;).
Mods are fairly simple but may offend purists as they bypass a lot of the dross in the OP code by the use of a couple of global variables.
The code has been flown on EasyStar II, Radian and Phantom Wing. The common feature of these is they are all foam and OK for this type of experiment. Having the Phantom return at high speed then orbiting in a high bank tight circle overhead is exciting.