[Paper Review] RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization

Zhecheng Yuan, Sizhe Yang+4 more|arXiv (Cornell University)|Jul 15, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

RL-ViGen is a new reinforcement learning benchmark for visual generalization that integrates diverse, realistic task environments—such as manipulation, navigation, and autonomous driving—across multiple generalization types including lighting, camera views, and visual appearances. Evaluated within a unified framework, state-of-the-art algorithms show no universal superiority, highlighting the need for more robust, generalizable agents in real-world settings.

ABSTRACT

Visual Reinforcement Learning (Visual RL), coupled with high-dimensional observations, has consistently confronted the long-standing challenge of out-of-distribution generalization. Despite the focus on algorithms aimed at resolving visual generalization problems, we argue that the devil is in the existing benchmarks as they are restricted to isolated tasks and generalization categories, undermining a comprehensive evaluation of agents' visual generalization capabilities. To bridge this gap, we introduce RL-ViGen: a novel Reinforcement Learning Benchmark for Visual Generalization, which contains diverse tasks and a wide spectrum of generalization types, thereby facilitating the derivation of more reliable conclusions. Furthermore, RL-ViGen incorporates the latest generalization visual RL algorithms into a unified framework, under which the experiment results indicate that no single existing algorithm has prevailed universally across tasks. Our aspiration is that RL-ViGen will serve as a catalyst in this area, and lay a foundation for the future creation of universal visual generalization RL agents suitable for real-world scenarios. Access to our code and implemented algorithms is provided at https://gemcollector.github.io/RL-ViGen/.

Motivation & Objective

• Address the limitations of existing visual RL benchmarks, which are restricted to isolated tasks and narrow generalization categories.
• Overcome the risk of algorithmic overfitting to current benchmarks by introducing a broader, more diverse evaluation framework.
• Enable comprehensive, fair, and reliable evaluation of visual generalization capabilities across multiple task categories and visual perturbations.
• Provide a unified training and evaluation framework for state-of-the-art visual RL generalization algorithms to ensure consistent comparison.
• Lay the foundation for developing universal visual generalization agents suitable for real-world deployment.

Proposed method

• Design a benchmark with five task categories: locomotion, table-top manipulation, autonomous driving, indoor navigation, and dexterous hand manipulation.
• Incorporate multiple generalization types per task, including visual appearances, lighting changes, camera views, scene structures, and cross-embodiment transfer.
• Implement a unified training framework for various state-of-the-art visual RL algorithms, ensuring consistent optimization schemes and fair comparison.
• Use high-fidelity, realistic rendering for all environments to better reflect real-world visual variability.
• Integrate diverse data augmentation techniques (e.g., random crop, color jitter, saliency-based masking) to test robustness under visual perturbations.
• Evaluate performance using both generalization scores and training efficiency metrics (e.g., sample efficiency, wall-clock time).

Experimental results

Research questions

• RQ1 Can existing visual RL generalization algorithms achieve consistent performance across diverse, realistic task environments?
• RQ2 How do different generalization types—such as lighting changes, camera view shifts, and visual appearance variations—affect algorithm performance?
• RQ3 To what extent do current algorithms generalize across different embodiments and task structures?
• RQ4 What is the trade-off between generalization performance and training efficiency (e.g., sample efficiency, wall-clock time) across different algorithms?
• RQ5 Does the choice of data augmentation strategy significantly impact generalization in complex, high-dimensional visual environments?

Key findings

• No single algorithm consistently outperforms others across all tasks and generalization types, indicating that current methods lack universal generalization capability.
• DrQ-v2 demonstrates superior sample efficiency in locomotion and table-top manipulation tasks, particularly when agents are centrally positioned and subject to data augmentation noise.
• CURL achieves comparable sample efficiency to DrQ-v2 in locomotion and manipulation tasks, especially when using a single shared encoder with contrastive loss.
• Algorithms like SGQN and PIE-G show better generalization in complex visual scenarios such as dynamic video backgrounds and distractors, while DrQ-v2 struggles with entropy collapse in high-distraction settings.
• Wall-clock training time analysis shows DrQ-v2 is the most efficient, while methods using saliency maps (SGQN) or pre-trained representations (PIE-G) incur higher computational costs.
• The benchmark reveals that first-person view environments (e.g., Habitat, CARLA) are more robust to data augmentation noise, reducing performance variance across algorithms.