[Paper Review] Learning Dynamics Models for Model Predictive Agents

Michael Lutter, Leonard Hasenclever+6 more|arXiv (Cornell University)|Sep 29, 2021

Reinforcement Learning in Robotics54 references4 citations

TL;DR

This paper systematically evaluates design choices for learning dynamics models in model predictive control, using supervised training on data from a model-free agent in five DeepMind Control Suite environments. It finds that mean squared error is a poor predictor of planning performance, and identifies optimal configurations—such as multi-step loss with ensembles for deterministic models and input noise with large ensembles for stochastic models—enabling high-reward, stable planning close to ground-truth performance.

ABSTRACT

Model-Based Reinforcement Learning involves learning a extit{dynamics model} from data, and then using this model to optimise behaviour, most often with an online extit{planner}. Much of the recent research along these lines presents a particular set of design choices, involving problem definition, model learning and planning. Given the multiple contributions, it is difficult to evaluate the effects of each. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model -- the simulator. First, we collect a rich dataset from the training sequence of a model-free agent on 5 domains of the DeepMind Control Suite. Second, we train feed-forward dynamics models in a supervised fashion, and evaluate planner performance while varying and analysing different model design choices, including ensembling, stochasticity, multi-step training and timestep size. Besides the quantitative analysis, we describe a set of qualitative findings, rules of thumb, and future research directions for planning with learned dynamics models. Videos of the results are available at https://sites.google.com/view/learning-better-models.

Motivation & Objective

• To isolate and evaluate the impact of individual design choices in learning dynamics models for model predictive control.
• To identify best practices for training feed-forward neural network dynamics models that enable high-performing planning.
• To assess whether model accuracy (e.g., MSE) correlates with actual planning performance in complex control tasks.
• To investigate failure modes such as model exploitation and instability in high-dimensional systems like the humanoid.
• To provide actionable guidelines for model learning in model-based reinforcement learning, especially for real-world deployment.

Proposed method

• Collected a rich dataset from a pre-trained model-free agent across five continuous control environments in the DeepMind Control Suite.
• Trained feed-forward neural networks in a supervised manner to predict state transitions using state and action as input.
• Evaluated planner performance under varying design choices: deterministic vs. stochastic models, 1-step vs. multi-step loss, ensemble usage, and input noise augmentation.
• Used model predictive control (MPC) with a fixed planning horizon to assess performance, comparing learned models against the ground-truth simulator.
• Quantified planning performance via expected and achieved cumulative rewards, and analyzed reward discrepancy to detect model exploitation.
• Conducted qualitative analysis of open-loop rollouts to assess prediction fidelity and stability over long horizons.

Experimental results

Research questions

• RQ1 How do different model learning design choices—such as stochasticity, multi-step loss, ensembling, and input noise—affect planning performance in model predictive control?
• RQ2 To what extent does mean squared error (MSE) correlate with actual planning performance when using learned dynamics models?
• RQ3 Why do certain models fail on high-dimensional systems like the humanoid, even when they perform well on simpler tasks?
• RQ4 What role does input noise play in improving model robustness, particularly for stochastic models?
• RQ5 Can model exploitation be detected and mitigated through analysis of reward discrepancy between expected and actual returns?

Key findings

• Mean squared error (MSE), whether 1-step or multi-step, is not a reliable predictor of planning performance, as models with higher MSE can still achieve high rewards.
• For deterministic models, combining multi-step loss with model ensembles is essential to achieve high and stable planning performance.
• Stochastic models require both large ensemble sizes and input noise during training to achieve high reward, with input noise being critical for tasks like ball in cup.
• The humanoid environment consistently fails across all learned models, producing divergent open-loop trajectories and highly over-optimistic reward estimates.
• Model exploitation is prevalent, especially in systems with strong nonlinearities (e.g., contact dynamics), where planners overestimate future rewards due to inaccurate modeling near discontinuities.
• Despite high MSE, some models—especially with coarse time steps—achieve high rewards, indicating that precise trajectory modeling is less critical than ensuring structural correctness (e.g., preventing tunneling in ball-in-cup).