[Paper Review] Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks.

Alexander F. Sax, Bradley Emi+4 more|arXiv (Cornell University)|Dec 31, 2018

Reinforcement Learning in Robotics15 citations

TL;DR

This paper proposes using mid-level visual representations—such as scene parsing and object detection—as a perception module to improve sample efficiency and generalization in deep reinforcement learning for active robotic tasks. By integrating these intermediate features, agents learn faster and generalize better than from-scratch training, especially in unseen environments, provided features are carefully selected for each task.

ABSTRACT

One of the ultimate promises of computer is to help robotic agents perform active tasks, like delivering packages or doing household chores. However, the conventional approach to solving vision is to define a set of offline recognition problems (e.g. object detection) and solve those first. This approach faces a challenge from the recent rise of Deep Reinforcement Learning frameworks that learn active tasks from scratch using images as input. This poses a set of fundamental questions: what is the role of computer if everything can be learned from scratch? Could intermediate tasks actually be useful for performing arbitrary downstream active tasks? We show that proper use of mid-level perception confers significant advantages over training from scratch. We implement a perception module as a set of mid-level visual representations and demonstrate that learning active tasks with mid-level features is significantly more sample-efficient than scratch and able to generalize in situations where the from-scratch approach fails. However, we show that realizing these gains requires careful selection of the particular mid-level features for each downstream task. Finally, we put forth a simple and efficient perception module based on the results of our study, which can be adopted as a rather generic perception module for active frameworks.

Motivation & Objective

• To investigate whether mid-level visual representations can improve sample efficiency and generalization in reinforcement learning for active robotic tasks.
• To address the fundamental question of whether intermediate perception modules are beneficial when agents can learn directly from pixels.
• To identify which mid-level features are most effective for specific downstream active tasks.
• To develop a simple, efficient, and generic perception module based on empirical findings for use in active vision frameworks.

Proposed method

• Designing a perception module that extracts mid-level visual representations such as semantic segmentation, object detection, and scene parsing from raw images.
• Integrating these mid-level features as input to a deep reinforcement learning agent instead of raw pixels.
• Training the agent on a variety of active tasks (e.g., navigation, object manipulation) using the mid-level features as observations.
• Comparing performance against a baseline agent trained from scratch on raw pixels, measuring sample efficiency and generalization across environments.
• Systematically evaluating different combinations of mid-level features to identify the most effective set for each task.
• Proposing a lightweight, generic perception module based on the most effective features identified in experiments.

Experimental results

Research questions

• RQ1 Can mid-level visual representations improve sample efficiency in learning active tasks via deep reinforcement learning?
• RQ2 Does using mid-level features enhance generalization to unseen environments compared to training from raw pixels?
• RQ3 Which specific mid-level features are most beneficial for different downstream active tasks?
• RQ4 Is the performance gain from mid-level features dependent on careful feature selection, or is any intermediate representation sufficient?

Key findings

• Learning with mid-level visual representations leads to significantly higher sample efficiency than training from scratch.
• Agents using mid-level features generalize better to unseen environments, whereas from-scratch agents often fail in such settings.
• The performance gains are highly dependent on selecting task-specific mid-level features; not all features provide equal benefit.
• A simple, generic perception module based on the most effective features was successfully developed and shown to be effective across multiple tasks.
• The study demonstrates that intermediate perception can be a powerful complement to end-to-end learning in active vision systems.