[Paper Review] Associating Grasping with Convolutional Neural Network Features.

Li Yang Ku, Erik Learned-Miller+1 more|arXiv (Cornell University)|Sep 13, 2016

Robot Manipulation and Learning22 references3 citations

TL;DR

This paper proposes a CNN-based method that maps visual input to robot hand pre-shaping for grasping by leveraging pre-trained features and targeted backpropagation to localize manipulable object structures. By identifying 3D positions of hierarchical features across CNN layers, the approach enables effective pre-shaping in cluttered scenes and outperforms point cloud-based methods on Robonaut-2.

ABSTRACT

In this work, we provide a solution for pre-shaping a human-like robot hand for grasping based on visual information. Our approach uses convolutional neural networks (CNNs) to define a mapping between images and grasps. Applying CNNs to robotics applications is non-trivial for two reasons. First, collecting enough robot data to train a CNN at the same scale as the models trained in the vision community is extremely difficult. In this work, we demonstrate that by using a pre-trained CNN, a small set of grasping examples is sufficient for generalizing across different objects of similar shapes. Second, the final output of a CNN contains little location information of the observed object, which is essential for the robot to manipulate the object. We take advantage of the hierarchical nature of CNN layers and identify the 3D positions of features that capture the hierarchical support relations between filters in different CNN layers using an approach we call targeted backpropagation. Targeted backpropagation traces the activation of higher level features in a CNN backwards through the network to discover the locations in the observation that were responsible for making them fire, thus localizing important structures that are manipulable in the environment. We show that this approach outperforms approaches without targeted backpropagation in a cluttered scene. We further implement a hierarchical controller that controls fingers and palms based on features located in different layers of the CNN for pre-shaping the robot hand and demonstrate that this approach outperforms a point cloud based approach on a grasping task on Robonaut-2.

Motivation & Objective

• To address the challenge of generalizing robot grasping across object shapes using limited real-world training data.
• To overcome the lack of spatial localization in standard CNN outputs, which hinders precise robotic manipulation.
• To enable effective pre-shaping of a human-like robot hand using hierarchical CNN features with spatial localization.
• To demonstrate improved grasping performance in cluttered environments compared to baseline methods.

Proposed method

• Utilizes a pre-trained convolutional neural network (CNN) to extract visual features from images without requiring large-scale robot-specific datasets.
• Applies targeted backpropagation to trace activations from higher-level CNN features back to their spatial origins in the input image, identifying key manipulable structures.
• Identifies 3D positions of feature detectors across multiple CNN layers to capture hierarchical support relations between filters.
• Constructs a hierarchical controller that maps features from different CNN layers to control individual fingers and the palm of the robot hand.
• Uses the localized feature maps to guide pre-shaping of the robot hand before contact, ensuring alignment with object geometry.
• Employs a multi-scale feature representation to improve robustness in cluttered scenes where object boundaries are ambiguous.

Experimental results

Research questions

• RQ1 Can a small number of grasping demonstrations generalize across similar object shapes when using a pre-trained CNN?
• RQ2 How can spatial localization of CNN features be achieved to enable precise robotic manipulation?
• RQ3 Does targeted backpropagation improve grasp success in cluttered environments compared to standard CNN feature maps?
• RQ4 Can hierarchical control based on multi-layer CNN features outperform point cloud-based approaches in pre-shaping?
• RQ5 How does the integration of hierarchical feature localization enhance grasping performance on a humanoid robot like Robonaut-2?

Key findings

• The method achieves effective pre-shaping using only a small number of grasping demonstrations, demonstrating generalization across similar object shapes.
• Targeted backpropagation successfully localizes relevant object structures in the image, enabling precise hand configuration.
• The approach outperforms standard CNN feature maps in cluttered scenes due to improved spatial awareness of grasp-relevant features.
• The hierarchical controller based on multi-layer CNN features achieves higher grasping success than a point cloud-based baseline on Robonaut-2.
• The use of pre-trained CNNs with targeted backpropagation reduces the need for large-scale robot training data while maintaining high performance.