#PhINODE

## abstract

Fixed-wing UAVs are efficient for long-range missions, but their attitude dynamics grow strongly nonlinear and coupled away from level flight, making them harder to control than the quadrotors that most of the literature targets. Reinforcement learning can learn controllers directly from data, but model-free RL requires a large number of costly simulator interactions. Model-based RL instead learns a dynamics model to train on, which can be made more reliable by embedding known aerodynamic physics as a prior. However, this assumes the prior is accurate, which is rarely the case. In this work we extend PhIHP, a physics-informed model-based RL method validated only on classic control benchmarks, to fixed-wing UAV attitude control. These test environments have dynamics that are low-dimensional and exactly known, and we test whether the approach holds on a system whose aerodynamics are neither. We build a physics prior for a simulated Skywalker X8 fixed-wing UAV, paired with a learned residual that corrects what the prior misses. We evaluate controllers on settled attitude tracking error: the steady-state deviation, in degrees, between commanded and achieved roll and pitch.

## in short

The equations of flight are known. The coefficients that go into them, for a new or lightweight airframe, are only approximate. That gap is the whole subject.

1. 1Train the controller inside a physics-informed model rather than in the simulator. The model is cheap, so it can run in parallel; the simulator cannot.
2. 2Correct the physics when it is inaccurate by letting the optimiser tune the prior's coefficients together with the learned residual.
3. 3Plan on top of the controller with a short-horizon MPC that uses the trained policy and its critic, and find out which of the two actually helps.

## background

Two ways to build a controller

Planning, as in model predictive control, needs a model of the dynamics and optimises actions over a short horizon at every step. Reinforcement learning needs no model and learns a policy from interaction, but pays for it in samples: model-free methods need a great many. Model-based RL sits between the two: learn a model of the dynamics, then train the policy inside it. This work combines all three, a learned model, a policy trained in it, and a planner on top.

Reinforcement learning

An agent learns a policy π that maximises the expected discounted return, E_π[Σ_t γ^t r_t], through trial and error in an environment formalised as a Markov decision process (S, A, T, r, γ): states, actions, transition dynamics, reward and a discount factor. Model-free algorithms come in three families: value-based (learn Q(s, a) and act on it), policy-based (optimise π directly by policy gradient), and actor-critic, which does both, an actor proposing actions and a critic evaluating them, and reduces gradient variance considerably.

TD3

The controller here is TD3, an off-policy actor-critic for continuous actions. The actor is a deterministic policy π_φ(o); two independent critics Q_θ1 and Q_θ2 are trained and the smaller of the two forms the target, which is what keeps the value from being overestimated:

The clipped noise ε smooths the target policy so the critics cannot exploit sharp, incorrect peaks in the estimated value, and the actor is updated only once every few critic updates, following the gradient of the first critic. Slowly updated target networks θ' and φ' stabilise the whole thing. The critic Q is also what makes the hybrid controller work later on.

Partial observability

A flight simulator keeps a far richer internal state than the agent is shown: position, altitude, propulsion variables the attitude task never exposes. The agent acts on an observation o_t, not the full state s_t, so the environment is formally a partially observable MDP, (S, A, O, T, Ω, r, γ), with an observation function Ω(o_t | s_t, a_t). The dynamics model and the policy are both built on that partial view.

environment: JSBSim through FW-JSBGym · Skywalker X8 · windless, turbulence-free, constant airflow, so the difficulty is the aircraft's own dynamics · task: drive roll and pitch to a commanded reference and hold it

## method

The dynamics model

Following APHYNITY and PhIHP, the state derivative is a known physics term plus a learned correction, integrated with RK4. The prior is approximate by design, so its coefficients θ_p are trainable.

The prior is an aerodynamic model of the Skywalker X8. It is degraded on purpose for the experiments: ten coefficients that are hard to measure in practice are perturbed by 25 to 50% (mild) or 75 to 100% (severe), and each condition is run with the prior frozen or trainable.

The hybrid controller

TD3 trained entirely in imagination gives a policy π and a critic Q. At inference, a CEM-MPC planner searches short action sequences through the dynamics model. The policy seeds the search; the critic scores what lies beyond the planning horizon.

setup: JSBSim full physics · no wind · settled error in degrees, < 1° is near-perfect · 10 seeds per condition · easy and hard target split · 10 ms inference budget

## results

q1Can the controller be trained inside the model alone?

Yes. A policy trained purely on trajectories imagined through the dynamics model, with no further simulator interaction, comes close to one trained directly on the simulator. Because the model is cheap to instantiate it can be parallelised, which the simulator does not easily allow: the policy converges faster and ends up best overall.

q2What if the physics prior is wrong?

With an accurate prior, tuning changes little. With a degraded one, tracking error rises sharply in non-nominal conditions, and letting the optimiser correct the prior's coefficients alongside the residual recovers most of that loss. On the hardest targets, 37 to 38% of it. Approximate physics is enough to learn a good controller.

q3Does planning on top of the controller help, and which part does the work?

The hybrid controller cuts tracking error on hard targets by 37% across all prior conditions and 41% with the true prior, and up to 43% in the best case. The ablation is the interesting part: almost all of the gain comes from the critic bootstrapping value past the horizon. Seeding the planner with the policy's own proposals does not help, and on its own it is much worse than the policy alone.

controller	all six prior conditions			true prior only
	all	easy	hard	all	easy	hard
policy alone	1.17	0.59	1.91	0.97	0.55	1.55
mpc + π	19.57	3.74	41.44	19.76	4.79	40.49
mpc + Q	0.82	0.48	1.30	0.67	0.50	0.90
hybrid (π + Q)	0.77	0.44	1.21	0.64	0.43	0.92

IQM settled attitude error, degrees. cost of the hybrid: 21 ms per step against a 10 ms budget, which is the main open problem.

exampleA hard dive

Roll and pitch tracking for an aggressive dive and roll manoeuvre, commanded roll −58° and pitch −21°. Dashed red is the reference; the shaded band is ±5°. The policy alone reaches the targets but its angular rates grow into oscillation; the hybrid controller settles and stays there.

(a) policy only

(b) hybrid controller

## what's next

This is the starting point of the PhD at ONERA and ISIR. The open directions, in order of how much they are on my mind:

• Waypoint tracking. From holding one attitude to sequencing spatial targets, a longer-horizon and more compositional task.
• Inference cost. 21 ms per step is twice the budget. Action delay, executing short sequences instead of replanning every step, or MPPI instead of CEM.
• Joint learning of model and controller, and hierarchical control across the attitude and guidance loops.

the phd, on the main page →