[Paper Review] Decoupled Data Based Approach for Learning to Control Nonlinear Dynamical Systems

Ran Wang, Karthikeya S. Parunandi+3 more|arXiv (Cornell University)|Apr 17, 2019

Adaptive Dynamic Programming Control4 citations

TL;DR

This paper proposes a decoupled data-based control (D2C) algorithm for learning optimal control in unknown nonlinear stochastic dynamical systems with continuous state and action spaces. By separating open-loop trajectory optimization from closed-loop feedback control via linearization, D2C achieves near-optimal performance with significantly reduced training time and improved data efficiency compared to state-of-the-art deep reinforcement learning methods like DDPG.

ABSTRACT

This paper addresses the problem of learning the optimal control policy for a nonlinear stochastic dynamical system with continuous state space, continuous action space and unknown dynamics. This class of problems are typically addressed in stochastic adaptive control and reinforcement learning literature using model-based and model-free approaches respectively. Both methods rely on solving a dynamic programming problem, either directly or indirectly, for finding the optimal closed loop control policy. The inherent `curse of dimensionality' associated with dynamic programming method makes these approaches also computationally difficult. This paper proposes a novel decoupled data-based control (D2C) algorithm that addresses this problem using a decoupled, `open loop - closed loop', approach. First, an open-loop deterministic trajectory optimization problem is solved using a black-box simulation model of the dynamical system. Then, a closed loop control is developed around this open loop trajectory by linearization of the dynamics about this nominal trajectory. By virtue of linearization, a linear quadratic regulator based algorithm can be used for this closed loop control. We show that the performance of D2C algorithm is approximately optimal. Moreover, simulation performance suggests significant reduction in training time compared to other state of the art algorithms.

Motivation & Objective

• Address the challenge of learning optimal control policies for unknown nonlinear dynamical systems with continuous state and action spaces.
• Overcome the computational intractability of dynamic programming due to the 'curse of dimensionality' in high-dimensional systems.
• Develop a data-efficient, reliable, and scalable control framework that avoids the sample inefficiency and training instability common in deep reinforcement learning.
• Enable practical deployment by ensuring robustness to moderate system noise and predictable training behavior.
• Provide a systematic alternative to model-free RL that maintains performance while drastically reducing training time and hyperparameter tuning.

Proposed method

• First, solve an open-loop trajectory optimization problem using a black-box simulation model of the system, treating the dynamics as known for planning.
• Second, collect random input-output perturbation data around the nominal trajectory to identify a linear time-invariant (LTI) system model via system identification.
• Use the identified LTI model to design a linear quadratic regulator (LQR) for closed-loop feedback control around the nominal trajectory.
• Leverage linearization about the nominal trajectory to apply efficient LQR-based control synthesis, avoiding full dynamic programming.
• Decouple the planning (open-loop) and control (closed-loop) phases to reduce computational complexity and improve training stability.
• Ensure data efficiency by minimizing interaction with the true system, relying instead on simulation for planning and limited perturbation data for system identification.

Experimental results

Research questions

• RQ1 Can a decoupled approach separate trajectory planning from feedback control to improve data efficiency and training stability in unknown nonlinear systems?
• RQ2 To what extent does the D2C method achieve near-optimal performance in terms of control cost and convergence under model uncertainty?
• RQ3 How does D2C compare to state-of-the-art deep reinforcement learning algorithms like DDPG in terms of training time, data efficiency, and robustness?
• RQ4 What is the impact of system noise on D2C performance, and how does it compare to the robustness of model-free RL methods?
• RQ5 Can the D2C framework be scaled to higher-dimensional systems without requiring prior knowledge of the system model?

Key findings

• D2C achieves near-optimal performance, with theoretical justification showing it is optimal to second order in a noise parameter.
• Training time for D2C is drastically reduced—by up to 99%—compared to DDPG, with 3× to 4× faster training on 3-link and 6-link swimmer systems.
• In the 6-link swimmer system, D2C required only 9,489.3 seconds of training time, while DDPG required 88,160 seconds, despite DDPG being trained for longer durations.
• D2C demonstrated superior data efficiency, with successful policy learning even when DDPG failed to converge after extended training, particularly in higher-dimensional systems.
• D2C showed reliable and predictable training, avoiding the hyperparameter tuning and instability issues common in DDPG and other deep RL algorithms.
• The method remains robust to noise levels up to 100% of the maximum control signal, though performance degrades sharply beyond a threshold, indicating a need for improved feedback design under high uncertainty.