[Paper Review] Neural Object Descriptors for Multi-View Shape Reconstruction

Edgar Sucar, Kentaro Wada+1 more|arXiv (Cornell University)|Apr 9, 2020

Robotics and Sensor-Based Localization41 references9 citations

TL;DR

This paper proposes learnable, multi-class neural object descriptors integrated with a differentiable, probabilistic rendering engine to enable end-to-end, differentiable 3D shape reconstruction from single or multi-view RGB-D images. The framework achieves joint optimization of object shapes, poses, and camera trajectories, enabling accurate 3D reconstruction for applications like robot grasping, augmented reality, and the first object-level SLAM system with full differentiability.

ABSTRACT

The choice of scene representation is crucial in both the shape inference algorithms it requires and the smart applications it enables. We present efficient and optimisable multi-class learned object descriptors together with a novel probabilistic and differential rendering engine, for principled full object shape inference from one or more RGB-D images. Our framework allows for accurate and robust 3D object reconstruction which enables multiple applications including robot grasping and placing, augmented reality, and the first object-level SLAM system capable of optimising object poses and shapes jointly with camera trajectory.

Motivation & Objective

• To address the challenge of robust and accurate 3D object shape reconstruction from RGB-D images using a learnable, multi-class representation.
• To develop a differentiable and probabilistic rendering engine that supports end-to-end optimization of object shapes and camera trajectories.
• To enable joint optimization of object poses, shapes, and camera trajectory, facilitating principled 3D reconstruction.
• To support practical applications such as robot grasping, augmented reality, and object-level SLAM.
• To provide a scalable and efficient framework for multi-view 3D reconstruction using deep learning and differentiable rendering.

Proposed method

• The framework employs multi-class neural object descriptors that are trained to represent diverse object shapes efficiently.
• A novel differentiable and probabilistic rendering engine simulates RGB-D observations from 3D object hypotheses, enabling gradient-based optimization.
• The system performs end-to-end optimization by backpropagating through the rendering process to refine object shapes, poses, and camera trajectories.
• Object descriptors are shared across classes, enabling generalization and efficient inference across multiple object categories.
• The method supports both single-view and multi-view input, leveraging geometric consistency across views for improved reconstruction accuracy.
• The differentiable rendering engine models uncertainty in observations, enabling principled probabilistic inference of 3D shapes.

Experimental results

Research questions

• RQ1 Can learned neural object descriptors enable accurate and generalizable 3D shape reconstruction from RGB-D images?
• RQ2 Can a differentiable and probabilistic rendering engine support end-to-end optimization of object shapes and camera trajectories?
• RQ3 Can joint optimization of object shapes, poses, and camera trajectory improve 3D reconstruction accuracy compared to sequential or independent optimization?
• RQ4 Can the framework support real-world applications such as robot grasping and augmented reality?
• RQ5 Is it feasible to build an object-level SLAM system that jointly optimizes object geometry and scene structure?

Key findings

• The framework enables accurate 3D reconstruction of objects from one or more RGB-D images using differentiable and probabilistic rendering.
• Joint optimization of object shapes, poses, and camera trajectory leads to more consistent and accurate reconstructions than independent optimization.
• The system supports the first object-level SLAM system capable of end-to-end differentiable optimization of object geometry and scene structure.
• The learned object descriptors generalize across object classes, enabling multi-class 3D reconstruction with shared representation.
• The approach enables practical applications such as robot grasping and augmented reality due to high-fidelity 3D shape estimation.
• The differentiable rendering engine supports uncertainty-aware inference, improving robustness to noisy or incomplete observations.