[논문 리뷰] A Study on Overfitting in Deep Reinforcement Learning

Chiyuan Zhang, Oriol Vinyals외 2명|arXiv (Cornell University)|2018. 04. 18.

Reinforcement Learning in Robotics참고 문헌 18인용 수 237

한 줄 요약

이 논문은 심층 강화학습에서의 과적합을 체계적으로 분석하여, 에이전트가 학습 미로를 암기하고 최적의 학습 보상에도 불구하고 테스트 성능이 큰 차이를 보일 수 있으며, 일반적인 확률 기반 방법은 과적합을 탐지하거나 예방하지 못할 수 있음을 보여준다.

ABSTRACT

Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learning, more training power comes with a potential risk of more overfitting. As deep RL techniques are being applied to critical problems such as healthcare and finance, it is important to understand the generalization behaviors of the trained agents. In this paper, we conduct a systematic study of standard RL agents and find that they could overfit in various ways. Moreover, overfitting could happen "robustly": commonly used techniques in RL that add stochasticity do not necessarily prevent or detect overfitting. In particular, the same agents and learning algorithms could have drastically different test performance, even when all of them achieve optimal rewards during training. The observations call for more principled and careful evaluation protocols in RL. We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.

연구 동기 및 목표

• Investigate how deep RL agents generalize from training mazes to unseen mazes under varying difficulty and training data.
• Assess whether standard RL regularization techniques prevent overfitting or merely mask it during evaluation.
• Characterize memorization capacity of neural networks in RL when faced with randomized reward structures.
• Explore the role of inductive bias (network architecture) in generalization performance across regular and randomized tasks.

제안 방법

• Use an asynchronous A3C framework with a dedicated test worker to separate training and testing environments.
• Employ a configurable gridworld maze with BASIC, BLOCKS, and TUNNEL variants to control task difficulty and regularity.
• Introduce randomized reward perturbations in training mazes to measure memorization and generalization under noise.
• Evaluate overfitting by comparing training vs. test episode rewards across different training set sizes and maze difficulties.
• Compare MLP and ConvNet architectures to study inductive bias effects on memorization and generalization.
• Test regularization techniques (random starts, sticky actions, RAND-SPAWN) as both training regularizers and evaluation add-ons.]
• research_questions:[
• To what extent can deep RL agents memorize random maze configurations, and how does this memorization affect test performance?
• Do common stochasticity-based evaluation or regularization techniques reliably detect or prevent overfitting in deep RL?
• How does inductive bias, via architecture (MLP vs ConvNet) and task regularity, influence generalization in deep RL?
• How do training set size and maze difficulty impact the gap between training and testing performance in deep RL?
• What framework or protocols are needed to standardize RL generalization evaluation to identify overfitting?

실험 결과

연구 질문

• RQ1 To what extent can deep RL agents memorize random maze configurations, and how does this memorization affect test performance?
• RQ2 Do common stochasticity-based evaluation or regularization techniques reliably detect or prevent overfitting in deep RL?
• RQ3 How does inductive bias, via architecture (MLP vs ConvNet) and task regularity, influence generalization in deep RL?
• RQ4 How do training set size and maze difficulty impact the gap between training and testing performance in deep RL?
• RQ5 What framework or protocols are needed to standardize RL generalization evaluation to identify overfitting?

주요 결과

• Agents can memorize a large collection of training mazes, leading to drastic differences between training and test performance even when training rewards are optimal.
• Adding stochasticity during evaluation or regularization does not reliably prevent or detect overfitting in deep RL in randomized mazes.
• Test performance degrades with increased maze difficulty and smaller training sets, while training rewards still reach near-optimal values.
• ConvNets tend to generalize better than MLPs on regular, spatially invariant tasks, while memories can be formed for random tasks with sufficient capacity.
• Memorization capacity persists even under randomized rewards, causing high training performance but weak test generalization in many setups.