[Paper Review] Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning

Ryan Julian, Benjamin Swanson+4 more|arXiv (Cornell University)|Apr 21, 2020

Reinforcement Learning in RoboticsComputer Science65 references41 citations

TL;DR

The paper shows that fine-tuning pre-trained, off-policy RL policies can adapt vision-based robotic grasping to new backgrounds, objects, lighting, and morphologies using less than 0.2% of data required for training from scratch, outperforming ImageNet-based pre-training.

ABSTRACT

One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data. We also find that these positive results hold in a limited continual learning setting, in which we repeatedly fine-tune a single lineage of policies using data from a succession of new tasks. Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique fine-tuning experiments on a real robotic grasping system pre-trained on 580,000 grasps.

Motivation & Objective

• Demonstrate how to adapt vision-based robotic manipulation policies to new variations via fine-tuning with off-policy RL.
• Quantify data efficiency and performance gains from fine-tuning versus training from scratch or using ImageNet features.
• Evaluate robustness of pre-trained policies across diverse environmental and morphological changes.
• Investigate continual learning by repeatedly fine-tuning a single policy on successive tasks.

Proposed method

• Pre-train a vision-based grasping policy (QT-Opt) on 580,000 real grasp attempts across diverse objects.
• Evaluate base policy under six challenging modifications (background, lighting, gripper shape, robot morphology, unseen transparent objects).
• Propose a simple offline fine-tuning procedure that initializes from the pre-trained policy and learns from a target-task dataset combined with base-task data.
• Collect offline exploration data for target task (up to 800 grasps) and update the policy with reduced learning rate using data from both base and target tasks.
• Assess performance after fine-tuning on target tasks and compare to Scratch and ImageNet baselines.
• Conduct continual learning experiments by sequentially fine-tuning on multiple tasks and measuring transfer and stability.

Experimental results

Research questions

• RQ1 How well can a pre-trained off-policy RL policy adapt to substantial task and environment changes with limited new data?
• RQ2 Is RL-based pre-training essential, or can supervised ImageNet pre-training suffice for rapid fine-tuning in robotics?
• RQ3 Can offline fine-tuning support continual learning with minimal performance degradation across task sequences?

Key findings

• Fine-tuning with off-policy RL yields substantial performance gains across all challenge tasks after relatively small data sets (as low as 25 exploration grasps).
• Fine-tuning with RL outperforms both Scratch (random start) and ImageNet pre-training baselines across tasks like Checkerboard Backing, Harsh Lighting, and Transparent Bottles.
• The method achieves near-state performance with as little as 0.2% of data required to train from scratch for the base task.
• In continual learning, sequential fine-tuning incurs only small performance penalties ( typically 4–7 percentage points) compared to single-step fine-tuning.
• Training with RL-based pre-training leads to larger parameter changes in image-processing layers than ImageNet-based pre-training, indicating effective adaptation for new sensorimotor tasks.