What does it do if you leave the normal PID active in the dive? Oscillate? Wobble around as if the servo isn't strong enough when you know it is strong enough? Come straight down and require more pull out pitch than normal? Is the new servo rated to have at least as much torque as the old servo (maybe the strength is marginal)?
I wish we had telemetry to look at to see what the servo is being asked to do. It could be something like a very jittery servo signal that is being smoothed too much by the new servo, or a PID oscillation with one (slower) servo and not the other (faster one) like an oscillation when you run quad ESC protocols that are too slow for your PIDs.
Just an observation: Measuring servo error with a fixed servo signal is not the same as measuring the heli leveling error with a stabilizing FC running. Any slop in the servo
or even the linkage is removed by the FC. Imagine a roll in manual mode with the stick released when it is level. Ignore inertia and assume that the only issue is slop. Imagine that with the slop the swash plate never comes all the way back to neutral and so you get a continuing slow roll. Now imagine what the servo does with the FC in Rate mode when you release the stick when the copter is level. It continues to drift past level and the FC sees that the rate is not zero and drives the servo
far enough past neutral to both counteract the slop and the inertia. Servo and linkage slop doesn't really matter because the FC takes the slop out.
I think I saw a slight circular roll/pitch oscillation in forward flight. I wonder if the FC is not well aligned to the frame or if the heli roll and pitch response is not well aligned to the frame. I would fly in Manual mode and test to see if pitch and roll flips are well aligned to the frame. If not, I would adjust whatever is needed to get the manual flips aligned well.
I might also be inclined to measure vibration with GCS and a USB cable attached and try to get a good rotor balance if you think it could use it.
Hope these ramblings help.