I suspect you know all about VtolPF and how it works. Yaw to heading and forward pitch for speed.
I haven't looked at the FWPF code in a while, but I recall this is how it works.
First: Don't stabilize the rudder!!! Fixed wing rudder/yaw should be manual, no rudder stabilization for any flight mode. If you have rudder stabilized, set it to manual for all flight modes. That may fix your problem.
Second: Scratch head about how Vtol configuration will have Rate stabilization on yaw and FW configuration will have manual on yaw/rudder. Considering that this has a vertical stab which would limit yaw spinning in Vtol configuration, what do you think about flying Vtol with manual yaw?
The problem with stabilizing the rudder on FW is that when you go to make say a left banked turn with aileron and elevator (Attitude mode does this too), that you are yawing left in that left turn. The stabilized rudder says "I see some left drift that I will correct" and applies right rudder, which badly messes up the turn.
After setting FW rudder stabilization to Manual, I suggest that you try to fly it in Attitude mode. When you get it flying that way, FW RTB should work.
FYI: FWPF assumes that the airplane climbs when you add power and descends when you reduce power, that is how it controls altitude.
There is a "max bank angle" in FWPF called RollLimit. To make a turn it banks the wings, that's all. It banks a certain number of bank degrees per degree of course error (default is 0.2 per 1.0 as I recall). It is flying in Attitude mode so when the nose drops, it automatically pulls some up elevator.
Airplanes (with dihedral) without ailerons, with rudder work fine with this.
Stabilization and control are generally the same thing inside the FC. Think of the stick as simply saying where to stabilize it.
I don't think you should mix rudder and aileron. To get what you need, you would need to do that at a low enough level so that when the FC stabilizes the bank, that it is applying rudder at the same time, and that gets back to what we were trying to avoid, especially when that mix needs to change based on whether it is currently a Vtol or a FW. Your roll should be just a roll and nothing else.
FWPF by default makes very wide turns, to avoid steep banking, dropping the nose and recovering. I wanted a tighter PostiionHold / RTB-at-base, so one of my changes is to automatically add some up when you bank, to avoid the nose drop and recovery.