[Paper Review] Models, Pixels, and Rewards: Evaluating Design Trade-offs in Visual Model-Based Reinforcement Learning

Mohammad Babaeizadeh, Mohammad Saffar+5 more|arXiv (Cornell University)|Dec 8, 2020

Reinforcement Learning in Robotics50 references4 citations

TL;DR

This paper evaluates design trade-offs in visual model-based reinforcement learning by comparing models that predict only rewards versus those that predict future observations (images). It finds that predicting images significantly improves task performance, and image prediction accuracy correlates more strongly with downstream performance than reward prediction accuracy, highlighting a critical role for visual modeling in exploration and planning efficiency.

ABSTRACT

Model-based reinforcement learning (MBRL) methods have shown strong sample efficiency and performance across a variety of tasks, including when faced with high-dimensional visual observations. These methods learn to predict the environment dynamics and expected reward from interaction and use this predictive model to plan and perform the task. However, MBRL methods vary in their fundamental design choices, and there is no strong consensus in the literature on how these design decisions affect performance. In this paper, we study a number of design decisions for the predictive model in visual MBRL algorithms, focusing specifically on methods that use a predictive model for planning. We find that a range of design decisions that are often considered crucial, such as the use of latent spaces, have little effect on task performance. A big exception to this finding is that predicting future observations (i.e., images) leads to significant task performance improvement compared to only predicting rewards. We also empirically find that image prediction accuracy, somewhat surprisingly, correlates more strongly with downstream task performance than reward prediction accuracy. We show how this phenomenon is related to exploration and how some of the lower-scoring models on standard benchmarks (that require exploration) will perform the same as the best-performing models when trained on the same training data. Simultaneously, in the absence of exploration, models that fit the data better usually perform better on the downstream task as well, but surprisingly, these are often not the same models that perform the best when learning and exploring from scratch. These findings suggest that performance and exploration place important and potentially contradictory requirements on the model.

Motivation & Objective

• To understand how fundamental design choices in visual model-based reinforcement learning (MBRL) affect performance.
• To evaluate the impact of predicting future observations (images) versus only predicting rewards on downstream task performance.
• To investigate whether latent space modeling or other architectural choices significantly influence performance.
• To analyze the interplay between model accuracy, planning, and exploration in MBRL agents.
• To provide a clear, controlled comparison of model designs independent of algorithmic confounds.

Proposed method

• The authors compare multiple MBRL model architectures that differ in whether they predict images or only rewards, and whether dynamics are modeled in latent or observation space.
• All models are trained and evaluated using a common planning algorithm (CEM) with fixed hyperparameters to isolate the effect of model design.
• The study uses a consistent training and evaluation protocol across multiple environments, focusing on control performance and prediction accuracy metrics.
• Models are evaluated on both standard benchmarks requiring exploration and on fixed training data to decouple performance from exploration dynamics.
• Image and reward prediction accuracy are measured and correlated with downstream task performance to assess predictive fidelity.
• The analysis includes ablation studies on model capacity, such as downsized architectures, to assess the impact of modeling capacity on performance.

Experimental results

Research questions

• RQ1 Does predicting future observations (images) lead to better task performance than predicting only rewards in visual MBRL?
• RQ2 How does image prediction accuracy compare to reward prediction accuracy in predicting downstream task performance?
• RQ3 To what extent do architectural choices like using latent spaces or autoregressive modeling affect performance?
• RQ4 How do performance and exploration place conflicting requirements on the predictive model?
• RQ5 Are models that fit the data better always the best performers when learning from scratch, or does exploration alter this relationship?

Key findings

• Predicting future observations (images) leads to significantly better task performance than predicting only rewards, even though the latter provides sufficient information for control.
• Image prediction accuracy correlates more strongly with downstream task performance than reward prediction accuracy, suggesting visual fidelity is a key proxy for planning quality.
• Models that perform poorly on standard benchmarks requiring exploration often match the performance of top models when trained on the same fixed data, indicating exploration dependence.
• In the absence of exploration, models with higher data-fitting capacity (better prediction accuracy) generally perform better, but these are not always the best-performing models when learning from scratch.
• Design choices such as using latent spaces or specific network architectures have little to no effect on final task performance, challenging assumptions about their necessity.
• The findings suggest that performance and exploration impose potentially contradictory requirements on the predictive model, with image prediction enabling better exploration and long-term planning.