[Paper Review] Observational Overfitting in Reinforcement Learning

Xingyou Song, Yiding Jiang+3 more|arXiv (Cornell University)|Dec 6, 2019

Reinforcement Learning in Robotics61 references17 citations

TL;DR

This paper introduces observational overfitting in reinforcement learning, where agents overfit to spurious features in observations rather than underlying MDP dynamics. By modifying only the observation space of a fixed MDP, the authors demonstrate that implicit regularization emerges in both linear (LQR) and nonlinear (MLP, CNN) policies, with overparametrization improving generalization—challenging existing SL-style generalization bounds in RL.

ABSTRACT

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Motivation & Objective

• To isolate and analyze observational overfitting as a distinct cause of poor generalization in RL, separate from dynamics or architecture confounds.
• To develop a synthetic benchmark framework that fixes MDP dynamics while varying only the observation space to study observational overfitting.
• To investigate whether implicit regularization occurs in RL under observational overfitting, especially with overparametrized policies.
• To evaluate the effectiveness of standard generalization metrics (e.g., norm, margin) in predicting RL generalization under observational overfitting.
• To challenge the applicability of supervised learning generalization bounds to RL by showing their looseness in observational overfitting settings.

Proposed method

• Construct a family of MDPs by modifying only the observation function of a fixed underlying MDP, preserving dynamics and reward structure.
• Use linear quadratic regulators (LQR) with linear policies to analytically study generalization under observational overfitting via gradient descent.
• Apply the framework to classic Gym environments (e.g., CartPole, LunarLander) using MLP and convolutional policies to study nonlinear settings.
• Train agents on varying observation distributions and evaluate zero-shot performance on unseen observation variants to measure generalization.
• Measure implicit regularization via weight norm dynamics and margin distributions across training, comparing with standard SL-style generalization bounds.
• Abate generalization metrics (e.g., spectral-Fro, distance from initialization) and test their correlation with test performance in observational overfitting regimes.

Experimental results

Research questions

• RQ1 Does observational overfitting occur when the underlying MDP dynamics remain fixed but observation distributions vary, leading to poor zero-shot generalization?
• RQ2 To what extent does implicit regularization emerge in RL under observational overfitting, particularly with overparametrized neural networks?
• RQ3 Can standard generalization metrics from supervised learning (e.g., weight norms, margin distributions) predict generalization performance in RL under observational overfitting?
• RQ4 How do architectural choices (e.g., MLP vs. CNN) and observation modality (e.g., saliency, occlusion) affect generalization in observational overfitting?
• RQ5 Do existing generalization bounds from supervised learning remain tight or informative in the context of observational overfitting in RL?

Key findings

• In the LQR setting with linear policies and gradient descent, a generalization gap necessarily exists under observational overfitting, even with exact optimization.
• For CoinRun, overparametrized MLPs generalize better under observational overfitting, even when MDP dynamics vary per level, suggesting implicit regularization supports generalization.
• Occluding the scoreboard in Sonic the Hedgehog improved test performance by 10% (from 1052 to 1141 for NatureCNN, 1130 to 1250 for IMPALA), showing that agents overfit to spurious visual cues.
• Raw weight norms increase monotonically during training and fail to predict generalization, indicating that standard SL-style norm-based bounds are too loose for RL.
• Margin distributions converge to a fixed form post-training, but their correlation with RL generalization is weaker than in SL, suggesting different inductive biases in RL.
• Spectral-Fro and distance-from-initialization metrics show better correlation with generalization than raw norms, but remain insufficient to fully explain overparametrization effects in RL.