[Paper Review] Investigating Compounding Prediction Errors in Learned Dynamics Models

Nathan Lambert, Kristofer S. J. Pister+1 more|arXiv (Cornell University)|Mar 17, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper investigates the root causes of compounding prediction errors in learned dynamics models used in model-based reinforcement learning. Through systematic experiments on simulated and real-world systems, it identifies that underlying system dynamics are the primary factor influencing long-horizon prediction error, with model architecture and training details playing a secondary role. The key contribution is a data-driven analysis showing that unstable dynamics dominate error growth, offering actionable insights for designing more robust predictive models in robotics and control.

ABSTRACT

Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.

Motivation & Objective

• To understand the root causes of compounding prediction errors in one-step learned dynamics models used in model-based reinforcement learning (MBRL).
• To isolate and quantify the impact of system dynamics, data distribution, and model training on long-horizon prediction accuracy.
• To provide actionable insights for designing better predictive models by identifying which system and modeling factors most influence error growth.
• To challenge the assumption that model architecture or training hyperparameters are the main drivers of error, instead highlighting system dynamics as the dominant factor.

Proposed method

• Conducted controlled experiments across multiple simulated and real-world robotic systems with varying dynamics, noise levels, and dimensionalities.
• Collected and evaluated data under consistent conditions to isolate the effects of system dynamics from data distribution and model training.
• Trained a range of one-step dynamics models, including deterministic and probabilistic neural networks, using standard deep learning pipelines.
• Measured prediction error over increasing prediction horizons (h) to analyze error compounding behavior.
• Used expectation-based trajectory propagation for probabilistic models to assess error growth under uncertainty.
• Analyzed error patterns across different systems to identify whether errors grow linearly, accelerate, or level off over time.

Experimental results

Research questions

• RQ1 What factors—system dynamics, data distribution, or model training—most significantly influence compounding prediction error in learned dynamics models?
• RQ2 How does the shape and magnitude of prediction error evolve over long horizons across different types of systems?
• RQ3 To what extent do changes in system noise or dimensionality affect long-term prediction accuracy?
• RQ4 Does error growth in long-horizon predictions eventually level off, and if so, under what conditions?
• RQ5 Can simple models (e.g., linear or deterministic networks) outperform complex ones in specific system regimes?

Key findings

• The underlying dynamics of a system are the strongest determinant of long-horizon prediction error, with unstable dynamics leading to rapid error compounding.
• Even with identical data and model architectures, systems with inherently unstable dynamics exhibit significantly higher error growth than stable ones.
• Prediction error often levels off after an initial horizon h, suggesting that beyond this point, error growth slows, though predictions are already unreliable for control.
• Reducing system noise from moderate to zero provides diminishing returns in prediction accuracy, indicating that noise is not the primary source of error.
• Simple models such as linear models and deterministic neural networks serve as strong baselines, especially in low-dimensional systems.
• Model choice and training hyperparameters have a secondary effect compared to the intrinsic properties of the system’s dynamics.