[Paper Review] The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement Learning

Harm van Seijen, Hadi Nekoei+2 more|arXiv (Cornell University)|Jul 7, 2020

Reinforcement Learning in Robotics19 references4 citations

TL;DR

This paper introduces the Local Change Adaptation (LoCA) regret, a new metric to evaluate model-based behavior in reinforcement learning by measuring how quickly an agent adapts to a local environmental change. The method reveals that even state-of-the-art agents like MuZero fall significantly short of optimal model-based adaptation, despite strong single-task sample efficiency, highlighting a critical gap in adaptability that current metrics fail to capture.

ABSTRACT

Deep model-based Reinforcement Learning (RL) has the potential to substantially improve the sample-efficiency of deep RL. While various challenges have long held it back, a number of papers have recently come out reporting success with deep model-based methods. This is a great development, but the lack of a consistent metric to evaluate such methods makes it difficult to compare various approaches. For example, the common single-task sample-efficiency metric conflates improvements due to model-based learning with various other aspects, such as representation learning, making it difficult to assess true progress on model-based RL. To address this, we introduce an experimental setup to evaluate model-based behavior of RL methods, inspired by work from neuroscience on detecting model-based behavior in humans and animals. Our metric based on this setup, the Local Change Adaptation (LoCA) regret, measures how quickly an RL method adapts to a local change in the environment. Our metric can identify model-based behavior, even if the method uses a poor representation and provides insight in how close a method's behavior is from optimal model-based behavior. We use our setup to evaluate the model-based behavior of MuZero on a variation of the classic Mountain Car task.

Motivation & Objective

• To address the lack of a consistent, behaviorally grounded metric to evaluate model-based reinforcement learning (RL) methods.
• To isolate and measure model-based behavior independently from other factors like representation learning or exploration.
• To provide a quantitative benchmark for how close an RL agent comes to optimal model-based adaptation.
• To demonstrate that high single-task sample efficiency does not imply strong adaptability to environmental changes.
• To guide future development of more adaptable, model-based RL agents for real-world applications.

Proposed method

• Design a two-task environment setup where tasks differ only in a small region of the state space, requiring distinct optimal policies.
• Define the LoCA regret as the cumulative regret incurred during adaptation after a local change from the first to the second task.
• Use a tabular baseline to establish an upper bound on ideal model-based performance, enabling comparison.
• Apply the LoCA regret metric to evaluate MuZero and Sarsa(λ) on a modified Mountain Car environment with varying hyperparameters.
• Measure adaptation speed by tracking regret over time after the environment change, with lower regret indicating faster, more model-based adaptation.
• Use the relative gain metric to compare performance against the optimal model-based baseline.

Experimental results

Research questions

• RQ1 How well can existing deep model-based RL methods adapt to a local environmental change, independent of their single-task sample efficiency?
• RQ2 To what extent does model-based behavior, as measured by adaptation speed, correlate with state-of-the-art performance on standard benchmarks?
• RQ3 What hyperparameters or internal design choices in methods like MuZero hinder their ability to adapt quickly to local changes?
• RQ4 Can the LoCA regret metric reliably distinguish model-based from model-free behavior, even when representations are poor?
• RQ5 How does the inclusion of on-policy components affect a method’s ability to adapt to environmental changes?

Key findings

• MuZero’s LoCA regret (5.93 with K=3, ε=0.1) is significantly higher than the optimal model-based baseline (1.0), indicating substantial room for improvement in adaptability.
• The best-performing MuZero configuration (ε=0.2) achieves a relative gain of 3.25, showing it is still far from optimal model-based behavior.
• Increasing the planning horizon K from 3 to 5 improves default regret but does not improve LoCA regret, suggesting limited benefit for adaptation speed.
• MuZero fails to converge for ε ≥ 0.6, indicating that exploration hyperparameters critically affect its ability to learn and adapt.
• Sarsa(λ) performs poorly on LoCA regret (12.55), confirming its model-free nature and lack of fast adaptation to changes.
• The results demonstrate that high single-task sample efficiency does not imply strong model-based adaptation, challenging assumptions about current SOTA methods.