I did a little research and at the beginning I tought you need various shapes but this
https://stackoverflow.com/questions/924171/geo-fencing-point-inside-outside-polygoncovers nearly everything. Even a circle can be fairly apprioximated with 6 or 8 edges.
I would define whether the UAV must stay out or has be inside, so you can have one polygon for the entire boundary and other polygons for objects inside these boundary.
If possible I would keep the altitude inside the requirements, but check, whether no intersections between polygons occurr.
If you have a tree or a house you can overfly it. You can also define max altitude and a minimum altitude if you want to avoid crashing into ground.
The code is in Github, right?
https://github.com/librepilot/LibrePilotI found the code for the pathplanner, but not for manual flight, but I only checked for one hour.
I also see no trouble to check whether the speed vector together with the braking deceleration is used for the point inside the geofence or the UAV itself, but we can use last for the start.
You also have to define a vector to escape the geofence area and for the pathplanner a solution / strategy to go arround the geofence and follow your flight plan. Only movements in the direction of this vector are allowed. For pathplanner I would look for the shortest way to the target and if the target is inside the geofence I would calculate the two points entering and leaving the geofence and find a solution to get from the enter to the exit point.
The geofence objects can then be updated by any other method. GCS or separate arduino with sensors.
I am unsure whether the FC can handle the workload, but I found a queue for those kind of tasks. Is this used to guarantee flight maneuvers are real time?