Learning Anisotropic Value Geometry with Finsler Reinforcement Learning

FiRL learns locomotion policies with a direction-dependent Finsler cost \(F(x,v)\) and a dynamic CVaR objective. The framework captures uphill/downhill asymmetry, lateral slip, and rare high-cost failures within a single risk-sensitive reinforcement learning objective.

FiRL builds a local cost geometry for locomotion and then trains an actor--critic policy to minimize tail-risk under that geometry. The key idea is that moving in different directions can have different physical costs: climbing uphill is not the same as descending, and lateral slip is not the same as forward motion.

Direction-dependent Finsler cost:

FiRL defines the one-step locomotion cost using a Finsler metric \(F(x,v)\), where \(x\) is the robot state and \(v\) is the induced motion direction:

\[ F(x,v) = w_e F_{\mathrm{energy}}(x,v) + w_d F_{\mathrm{drift}}(x,v) + w_f F_{\mathrm{friction}}(x,v). \]

The energy term measures basic motion effort, the drift term penalizes uphill motion, and the friction term penalizes lateral movement or slip. This lets the robot distinguish between actions that may look similar in Euclidean space but have very different physical consequences.

Quasi-metric path geometry:

Because \(F(x,v)\) can assign different costs to \(v\) and \(-v\), the induced path cost is generally asymmetric. For two states \(x\) and \(y\), FiRL defines:

\[ d_F(x,y) = \inf_{\tau:x\to y} \int_0^1 F(\tau(t),\dot{\tau}(t))\,dt. \]

This path cost satisfies the triangle inequality through path concatenation, while still allowing \(d_F(x,y) \neq d_F(y,x)\). In this sense, FiRL gives the policy an asymmetric geometric bias for direction-aware locomotion.

Dynamic CVaR Bellman objective:

To reduce rare but costly failures, FiRL optimizes a dynamic CVaR objective rather than only expected cost. The Bellman update is:

\[ \begin{aligned} V_\alpha(x) &= \min_{u\in\mathcal{U}(x)} \Bigl\{ c(x,u) + \gamma\,\mathrm{CVaR}_{\alpha} \left[V_\alpha(X')\right] \Bigr\}, \\ &\qquad X'\sim P(\cdot\mid x,u), \end{aligned} \]

where \(c(x,u)=F(x,v(x,u))\). This objective encourages the robot to avoid actions that may lead to high-cost tail outcomes, such as slipping, falling, or using excessive energy.

Distributional critic and actor update:

FiRL uses a distributional critic \(Z_\phi(x)\) to estimate the future cost distribution. The critic outputs quantiles of the return distribution, and the CVaR target is computed from the worst-cost tail:

\[ y_i = c_i + \gamma(1-d_i) \widehat{\mathrm{CVaR}}_\alpha \left[Z_{\phi^-}(x'_i)\right]. \]

The actor is then updated using advantages derived from this CVaR-based backup. This keeps the optimization practical while preserving FiRL’s main structure: direction-dependent cost plus tail-risk-aware learning.

Policy behavior:

In slopes, wind, stairs, and narrow-support settings, FiRL learns to avoid direct but risky motions when safer alternatives exist. Instead of only maximizing average performance, it learns cautious, energy-aware behaviors that reduce high-cost failures.

We evaluate FiRL across simulation benchmarks and preliminary real-world robot trials. The main question is whether direction-dependent Finsler costs and CVaR risk optimization help robots move more safely and efficiently under slopes, lateral disturbances, contact-rich terrain, and noisy terrain estimates.

Across MuJoCo and Isaac Sim tasks, FiRL improves success rate, reduces normalized energy cost, and lowers worst-case tail risk compared with risk-neutral PPO and other risk-aware or geometry-aware baselines.

Table: Aggregate MuJoCo performance across SlopedHopper, Walker2d, and HalfCheetah tasks. Energy and CVaR risk are normalized relative to PPO. Higher success is better; lower energy and risk are better.

Energy-risk trade-off.
FiRL achieves the best trade-off between energy efficiency and tail-risk reduction. Standard CVaR-PPO lowers risk but often spends more energy, while PPO+Finsler improves energy but does not sufficiently reduce tail risk. FiRL combines both benefits.

---

High-Fidelity Quadruped Evaluation

We further evaluate FiRL on a Spot-like quadruped in Isaac Sim across three contact-rich terrains: ramp climbing, stair traversal, and narrow platform-beam crossing. These environments introduce richer dynamics, contact events, and failure modes than the MuJoCo tasks.

Table: Isaac Sim quadruped results over 100 evaluation episodes. FiRL lowers tail cost and fall rate while avoiding the energy penalty often seen in standard risk-aware baselines.

Robustness to terrain inclination.
As the slope becomes steeper, PPO suffers a larger drop in success rate and a higher tail-risk cost. FiRL remains more stable by using the Finsler cost to account for uphill effort and lateral motion.

---

Robustness to Perception Noise

FiRL uses local terrain information to compute the Finsler cost, such as the terrain normal and uphill direction. To test whether the method is overly dependent on perfect terrain estimates, we perturb the perceived terrain normal with Gaussian angular noise during evaluation.

Table: Perception-noise ablation averaged over 5 seeds and 100 evaluation episodes per seed. Noise is injected into the terrain-normal estimate used by the slope-dependent Finsler cost term.

Terrain-normal noise robustness.
FiRL degrades gradually under noisy slope estimates and maintains strong success rates even under moderate angular noise. This suggests that the learned policy is not overly sensitive to small errors in the geometric cost estimate.

---

Preliminary Real-World Robot Trials

To complement the simulation results, we also conduct preliminary trials on a physical Spot robot. These trials are qualitative rather than a full hardware benchmark, but they show that the learned FiRL behavior can be executed on real hardware in narrow-support and discrete-foothold settings.

Physical robot trials.
The Spot robot demonstrates cautious traversal behavior on narrow supports and discrete footholds. These trials provide early evidence for real-world deployability, while a full perception-to-control hardware benchmark remains future work.