[논문 리뷰] Ray Interference: a Source of Plateaus in Deep Reinforcement Learning

Tom Schaul, Diana Borsa외 2명|arXiv (Cornell University)|2019. 04. 25.

Reinforcement Learning in Robotics참고 문헌 34인용 수 39

한 줄 요약

The paper analyzes a learning-dynamics phenomenon in deep RL where coupling between data generation and shared function approximators causes negative interference, leading to performance plateaus (ray interference). It characterizes conditions, shows connection to saddle points, and discusses remedies.

ABSTRACT

Rather than proposing a new method, this paper investigates an issue present in existing learning algorithms. We study the learning dynamics of reinforcement learning (RL), specifically a characteristic coupling between learning and data generation that arises because RL agents control their future data distribution. In the presence of function approximation, this coupling can lead to a problematic type of 'ray interference', characterized by learning dynamics that sequentially traverse a number of performance plateaus, effectively constraining the agent to learn one thing at a time even when learning in parallel is better. We establish the conditions under which ray interference occurs, show its relation to saddle points and obtain the exact learning dynamics in a restricted setting. We characterize a number of its properties and discuss possible remedies.

연구 동기 및 목표

• Motivate and define ray interference as a learning-dynamics issue in RL with function approximation.
• Analyze a minimal two-context bandit setting to derive exact learning dynamics.
• Characterize when plateaus occur and how winner-take-all regions contribute to slow learning.
• Generalize the phenomenon to factored objectives and multiple components in RL.
• Discuss prevalence, detection, and potential remedies for ray interference.

제안 방법

• Model the simplest (K x n) bandit with on-policy gradient updates to derive exact continuous-time dynamics.
• Define interference via cosine similarity of component gradients and identify saddle points.
• Derive the gradient dynamics for a (2x2) bandit to show fixed points and plateaus near saddle points.
• Introduce the notion of plateaus via higher-order derivatives and characterize their basins of attraction.
• Generalize to factored objectives with coupled components and analyze conditions for plateaus and WTA regions.
• Compare RL coupling with supervised learning and off-policy variants to illustrate how coupling and interference drive plateaus.

실험 결과

연구 질문

• RQ1 Under what conditions do ray interference and plateaus arise in RL with shared function approximators?
• RQ2 How do interference between objective components and coupling between performance and learning progress interact to create plateaus?
• RQ3 Can ray interference be predicted or detected in simple models and generalized to broader RL settings?
• RQ4 What remedies reduce interference and decouple learning dynamics in practice?
• RQ5 How does ray interference scale with more components or with different representations?

주요 결과

• Ray interference occurs when negative interference between components and coupling to future data generation cause learning trajectories to pass near saddle points, creating slow plateaus.
• In a (2x2) bandit, the gradients show persistent negative interference, yielding fixed points at corners and saddles at (0,1) and (1,0).
• Plateaus occur along inflection points where the learning acceleration changes sign, and their flatness scales with the slope of the learning progress near those points.
• When using tabular representations or off-policy/supervised setups, ray interference is mitigated or eliminated, indicating coupling and interference are key ingredients.
• Increasing the number of components K can widen and intensify plateaus, and plateaus can lengthen exponentially with learning stages in fully interfering settings.
• Off-policy learning or datasets that disrupt current-policy data generation can reduce coupling and thereby alleviate plateaus.