[Paper Review] Self-Supervised Goal-Conditioned Pick and Place

Coline Devin, Payam Rowghanian+3 more|arXiv (Cornell University)|Aug 26, 2020

Robot Manipulation and Learning11 references4 citations

TL;DR

This paper proposes a self-supervised framework that learns pixel-wise object embeddings from unlabeled pick-and-place interactions, enabling goal-conditioned grasping and placing without human-annotated data. By using contrastive learning on wrist-cam, grasp-bin, and place-bin images, the model predicts optimal grasp and place locations that generalize to unseen objects and textures, achieving 70% accuracy on unseen texture classes.

ABSTRACT

Robots have the capability to collect large amounts of data autonomously by interacting with objects in the world. However, it is often not obvious \emph{how} to learning from autonomously collected data without human-labeled supervision. In this work we learn pixel-wise object representations from unsupervised pick and place data that generalize to new objects. We introduce a novel framework for using these representations in order to predict where to pick and where to place in order to match a goal image. Finally, we demonstrate the utility of our approach in a simulated grasping environment.

Motivation & Objective

• To develop a self-supervised method for learning visual object representations from unlabeled robotic interaction data.
• To enable goal-conditioned grasping and placing without human-annotated segmentation or CAD models.
• To learn a unified embedding space that supports both grasp and place prediction using only pixel-level supervision from robot interactions.
• To evaluate generalization to novel objects and textures not seen during training.
• To demonstrate the feasibility of using real-world robotic data for visual representation learning without privileged supervision.

Proposed method

• The model uses two fully convolutional ResNet-35 encoders: one for bin images (grasp and place) and one for wrist-cam images of the gripped object.
• Contrastive learning is applied to push embeddings of the grasp point, place point, and end-effector pixel to be similar while pushing other pixels apart.
• Negative sampling is performed using either full-image negatives or gamma-distributed sampling to avoid collapsing nearby pixels.
• The model is trained with a joint contrastive loss that includes both wrist-to-bin and bin-to-bin comparisons (Grasp:Place).
• Grasp and place positions are predicted by maximizing the dot product between the wrist-cam embedding and the bin embeddings at each pixel.
• The framework uses a simulated robotic environment with randomized lighting and textured objects from the Describable Textures dataset.

Experimental results

Research questions

• RQ1 Can pixel-wise object embeddings be learned from self-supervised robotic interactions without human annotations?
• RQ2 Can such embeddings generalize to novel objects and textures not seen during training?
• RQ3 Does contrastive learning on grasp, place, and wrist-cam images enable accurate goal-conditioned pick-and-place?
• RQ4 How does negative sampling strategy affect the quality of the learned embeddings?
• RQ5 Can a single unified embedding space support both grasp and place prediction in a goal-conditioned setting?

Key findings

• The model achieves 70% grasp and place accuracy on unseen texture classes, demonstrating strong zero-shot generalization.
• Training on 50 different textures instead of 10 leads to improved performance across all validation sets, including on seen textures.
• The combination of gamma-distributed negative sampling and Grasp:Place contrastive loss yields the best performance, outperforming baselines.
• Performance drops from 83% on the training set to 69% on the same-texture validation set, but remains stable at ~70% on unseen textures.
• The learned similarity metric is highly sensitive to spatial position within objects, enabling accurate selection of grasp and place points.
• The framework generalizes to novel objects and textures without requiring additional data collection or privileged information.