[Paper Review] Learning Locomotion Controllers for Walking Using Deep FBSDE

Bolun Dai, Virinchi Roy Surabhi+2 more|arXiv (Cornell University)|Jul 16, 2021

Robotic Locomotion and Control17 references4 citations

TL;DR

This paper proposes a deep forward-backward stochastic differential equation (FBSDE) controller using an LSTM-based neural network to solve optimal control problems for robotic locomotion, specifically applying it to a linear inverted pendulum model (LIPM) with state constraints. The method achieves real-time, high-frequency control with 6.67× faster computation than CVX-based optimization, enabling stable, cyclic walking over 20+ steps in simulation under noise and constraints.

ABSTRACT

In this paper, we propose a deep forward-backward stochastic differential equation (FBSDE) based control algorithm for locomotion tasks. We also include state constraints in the FBSDE formulation to impose stable walking solutions or other constraints that one may want to consider (e.g., energy). Our approach utilizes a deep neural network (i.e., LSTM) to solve, in general, high-dimensional Hamilton-Jacobi-Bellman (HJB) equation resulting from the stated optimal control problem. As compared to traditional methods, our proposed method provides a higher computational efficiency in real-time; thus yielding higher frequency implementation of the closed-loop controllers. The efficacy of our approach is shown on a linear inverted pendulum model (LIPM) for walking. Even though we are deploying a simplified model of walking, the methodology is applicable to generalized and complex models for walking and other control/optimization tasks in robotic systems. Simulation studies have been provided to show the effectiveness of the proposed methodology.

Motivation & Objective

• To develop a computationally efficient, real-time control method for robotic locomotion tasks using deep learning.
• To extend deep FBSDEs to handle hybrid dynamical systems such as walking robots, which involve discrete transitions (e.g., footstep changes).
• To incorporate state constraints (e.g., minimum CoM velocity) into the FBSDE framework to enforce desired dynamic behaviors.
• To enable generalization of a single-step policy to multi-step walking via state reinitialization in a local foot-attached frame.
• To demonstrate superior computational efficiency compared to traditional optimization-based methods like CVX in high-dimensional settings.

Proposed method

• Formulates the locomotion control problem as a stochastic optimal control task, leading to a Hamilton–Jacobi–Bellman (HJB) equation.
• Uses a deep neural network (LSTM) to approximate the solution of the HJB equation, enabling end-to-end learning of the value function and optimal control policy.
• Employs a forward-backward stochastic differential equation (FBSDE) framework to couple the state dynamics with the adjoint (costate) dynamics, allowing for backpropagation through time.
• Introduces state constraints into the FBSDE formulation using a penalty-based method to enforce bounds on CoM velocity, ensuring robustness and stability.
• Adapts the policy for multi-step walking by reinitializing the state to a local frame at each footstep, with dynamic step length based on terminal velocity.
• Trains the controller on a single step of walking in simulation and generalizes it to longer sequences via state reframe and velocity-based step adjustment.

Experimental results

Research questions

• RQ1 Can deep FBSDEs be effectively applied to hybrid dynamical systems such as bipedal walking with discrete footstep transitions?
• RQ2 How does the proposed deep FBSDE controller compare in computational efficiency to traditional optimization solvers like CVX in real-time control scenarios?
• RQ3 Can state constraints (e.g., minimum CoM velocity) be effectively embedded into the FBSDE framework to guide desired locomotion behavior?
• RQ4 To what extent can a single-step policy be generalized to multi-step walking using state reinitialization and dynamic step length adjustment?
• RQ5 How robust is the learned controller under process noise and model uncertainty in the LIPM framework?

Key findings

• The deep FBSDE controller achieves a 6.67× speedup in control computation compared to CVX-based optimization, with lower variance, making it suitable for high-frequency real-time control.
• The controller maintains stable, cyclic walking behavior over 20 steps in both noiseless and noisy LIPM simulations, with terminal velocities converging to a narrow band around the nominal value.
• The method successfully enforces a minimum CoM velocity constraint of 1.0 m/s, with all 64 sampled trajectories in the constrained case remaining within bounds, unlike the unconstrained case which frequently violated the limit.
• The learned policy generalizes effectively to multi-step walking by reinitializing states to a local foot-attached frame at each step, enabling dynamic step length adjustment based on terminal velocity.
• The controller demonstrates robustness to noise: even under noisy conditions, the terminal velocity remains tightly clustered around the nominal value, indicating resilience to disturbances.
• The approach is computationally scalable, with performance gains increasing significantly for higher-dimensional systems and longer preview horizons compared to traditional solvers.