[Paper Review] Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

Pete Florence, Lucas Manuelli+1 more|arXiv (Cornell University)|Jun 22, 2018

Robot Manipulation and Learning32 references99 citations

TL;DR

Dense Object Nets learn pixelwise dense visual descriptors in a self-supervised way to support manipulation across rigid and non-rigid objects, with multi-object and class-generalization capabilities, enabling grasping of specific points and cross-instance/class transfer.

ABSTRACT

What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.

Motivation & Objective

• Develop a self-supervised dense visual descriptor representation for manipulation that is task-agnostic and applicable to rigid and non-rigid objects.
• Enable fast learning (≈20 minutes) of dense descriptors for unseen objects and object classes.
• Create techniques for multi-object distinct descriptors and for descriptors that generalize across classes or remain instance-specific.
• Demonstrate manipulation tasks that use learned dense descriptors to grasp specific points on objects and transfer grasps across instances or classes.

Proposed method

• Use a dense pixelwise contrastive loss to train a descriptor mapping f that maps an RGB image to a W×H×D descriptor space.
• Construct matches and non-matches from RGBD videos via 3D dense reconstructions, enabling self-supervised learning without human labels.
• Apply object-centric masking derived from 3D change detection to focus training on objects and ignore backgrounds.
• Introduce background domain randomization and hard-negative scaling to improve cross-scene consistency and descriptor precision.
• Extend to multi-object dense descriptors using cross-object loss, direct multi-object within-scene training, and synthetic multi-object scene composition.

Experimental results

Research questions

• RQ1 Can dense pixelwise descriptors learned via self-supervision provide consistent object representations across viewpoints and deformations?
• RQ2 How can multi-object and class-generalization be achieved without compromising instance-specific precision in dense descriptors?
• RQ3 To what extent can learned dense descriptors enable manipulation tasks such as grasping specific points across configurations or transferring grasps across object instances or classes?

Key findings

• Descriptors are consistent across significant deformations for a wide variety of objects, including low-texture items.
• Cross-object loss and larger descriptor spaces enable distinct descriptor regions for multiple objects without sacrificing performance.
• Descriptors can generalize across classes (hats, shoes, mugs) and transfer grasps across unseen instances within a class.
• Distinct-object techniques enable grasping specific points on objects in clutter and across multiple object configurations.
• Background domain randomization and orientation randomization improve descriptor robustness, especially on small datasets.