I've been considering my rather big mixer enhancement and I think for your special case, it only needs a small piece of it; a 2nd mixer table that gets used with configurable flight mode switch positions (and some programmer stuff). One FMS position set up for hovering in one mixer where e.g. roll is differential thrust and another FMS position set up for airplane flight in the other mixer where e.g. roll is aileron.
This would be implemented as a mixer table offset. Normally zero in all FMS positions. I am guessing that you have 6 independent output channels, so set the offset to 0 for hover FMS and 6 for airplane FMS.
Forgive me for being pedantic. I would like to remind that the mixer table is output (servo / actuator) configuration. For each "servo" you get a configured amount of roll plus a configured amount of pitch plus ... (where the command can come from transmitter stick position or "FC doing the same thing with code").
I presume you are 100% Windows. Can you do your own builds? I use Linux and don't build Windows GCS. I may be able to do a double cross compile, but I haven't set that up in years.
If you are interested in this then I ask that you get it tuned and flying well independently in both modes. Your mixer table will be completely different for hover than for airplane. E.g. In hover, roll will be connected to differential thrust (+roll for one servo and -roll for the other servo). In airplane mode of course roll will be connected to the ailerons and not differential thrust. E.g. In hover you may choose to have rudder and elevator outputs (servos) enabled in the mixer, but all inputs set to zero (this should hold both to neutral servo regardless of stick positions). In airplane mode rudder and elevator outputs will be active and you may choose to do away with differential thrust or move it to yaw input. Etc.
Do away with any mixes or mods that try to get a compromise between hover and airplane. I assume you have 6 controlled channels. Two independent motors (differential thrust plus throttle), two independent ailerons (elevons), airplane rudder, and airplane elevator. Let me know if you must have more or different than that as that could make or break the mod. At 6+6=12 we will be using all mixers. If you have 4 aileron servos, they should be physically wired to be controlled as 2 servos; left elevon and right elevon.
Have TreatCustomCraftAs set to VTOL. Lock the wing into hover position and get that configured, tuned (PIDs) and flying really well in Attitude mode. It should be doable including hovering at any compass heading relative to the wind. After you get it flying really well, you might try AutoTune (initial PIDs not too high) and see what PIDs that gives and whether it feels better to you.
Save that/those configuration(s) for later reference.
Have TreatCustomCraftAs set to FixedWing. Lock the wing into airplane position and get that configured, tuned (PIDs) and flying really well in Attitude mode. I always start with full Manual mode, then Rate, then finally Attitude. Do not adjust a previous step such as changing control horn throw (Manual) when doing Rate or such as Rate settings when doing Atti. When you e.g. first try Rate, make the mental assumption that you will change right back to Manual. A good Rate tuning rule is set I and D to zero and get P adjusted to mild oscillation, then cut P in half (multiply by 0.45 actually) and then tune I up to oscillation and back off till oscillation is completely gone. Leave D zero.
https://en.wikipedia.org/wiki/PID_controller#Ziegler%E2%80%93Nichols_methodWhen done, the main things you have are a mixer table and set of PIDs for hover and a very different mixer table and set of PIDs for airplane. Different mixer tables will be handled with this mixer table offset enhancement. Different PIDs will be handled using current "stabilization settings banks" functionality.
Without a way of locking into airplane mode, I don't know if it's reasonable to consider waypoint missions.
Let me know if you want to try this, whether you can build a GCS, and what version you want to start from.
Firmware is the same wherever it is built, so I can do that much for sure. This version (GCS and firmware) should be considered incompatible with everything else although most stuff will work. StabilizationSettings is probably where I would put the mixer table offset (one per flight mode) and so it would be incompatible with all other versions.