[Paper Review] On-Robot Learning With Equivariant Models

Dian Wang, Mingxi Jia+3 more|arXiv (Cornell University)|Mar 9, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes On-Robot Learning with Equivariant SAC, a sample-efficient reinforcement learning method that leverages discrete symmetry groups (e.g., D₄, C₈) and data augmentation to train robotic manipulation policies directly on physical robots in under two hours. It demonstrates that pre-training in simulation is often unnecessary and can even cause negative transfer, making direct on-robot learning viable and efficient for simple manipulation tasks.

ABSTRACT

Recently, equivariant neural network models have been shown to improve sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which a policy must be learned entirely on a physical robotic system without reference to a model, a simulator, or an offline dataset. We focus on applications of Equivariant SAC to robotic manipulation and explore a number of variations of the algorithm. Ultimately, we demonstrate the ability to learn several non-trivial manipulation tasks completely through on-robot experiences in less than an hour or two of wall clock time.

Motivation & Objective

• To enable efficient on-robot policy learning without relying on simulators or offline datasets.
• To investigate whether equivariant models with discrete symmetry groups outperform continuous ones in robotic manipulation.
• To evaluate the impact of data augmentation on equivariant models trained directly on real robots.
• To assess whether sim2real pre-training is necessary or beneficial for on-robot learning.
• To understand when and why sim2real transfer fails, particularly due to domain shift in contact dynamics.

Proposed method

• Uses Equivariant Soft Actor-Critic (SAC) with discrete symmetry groups (e.g., D₄, C₈) to encode rotation and reflection invariances into the policy network.
• Applies irreducible representations of the group to enforce equivariance in the neural network architecture, improving sample efficiency.
• Employs data augmentation techniques—particularly buffer augmentation—on the replay buffer to further improve performance on discrete symmetry groups.
• Trains policies entirely on the real robot using real-world experience, without simulation or pre-training.
• Compares on-robot training with sim2real fine-tuning using PyBullet simulations to evaluate transfer effectiveness.
• Evaluates performance across four manipulation tasks: Block Picking, Clutter Grasping, Block Pushing, and Block in Bowl.

Experimental results

Research questions

• RQ1 Do discrete symmetry groups (e.g., D₄, C₈) lead to better performance than continuous groups (e.g., SO(2), O(2)) in on-robot reinforcement learning?
• RQ2 Does data augmentation improve the performance of equivariant models trained on real robots, despite the model already encoding symmetries?
• RQ3 Is sim2real pre-training necessary or beneficial for on-robot learning of manipulation policies?
• RQ4 Under what conditions does sim2real pre-training lead to negative transfer in real-world robotic policy learning?
• RQ5 Can equivariant models trained on real robots achieve high performance on complex manipulation tasks in under two hours of wall-clock time?

Key findings

• Equivariant SAC with discrete symmetry groups (D₄, C₈) outperformed continuous group equivariance (SO(2), O(2)) in all evaluated tasks, indicating better generalization despite approximation.
• Data augmentation, especially buffer augmentation, significantly improved performance on discrete symmetry groups, demonstrating that data augmentation remains beneficial even when symmetries are encoded in the model.
• On-robot learning with equivariant models achieved 92% success in Block Pushing and 92% in Block in Bowl, with 70% in Clutter Grasping, all within 1–2 hours of real-world training.
• Sim2real pre-training was not necessary for success, and in Block Pushing, it led to negative transfer, where the pre-trained policy performed worse than training from scratch due to differing contact dynamics in simulation.
• The sim2real agent learned to push down on the block (due to softer contact compliance in PyBullet), while the on-robot agent learned to push sideways, highlighting a domain shift that undermines transfer.
• In Block Picking and Block in Bowl, both on-robot and sim2real fine-tuning achieved 100% success, but on-robot training converged faster in some cases, showing that direct learning can be competitive.