[Paper Review] Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Stephan R. Richter, Stefan Roth|arXiv (Cornell University)|Apr 29, 2018

Advanced Vision and ImagingComputer Science1 references68 citations

TL;DR

The paper introduces Matryoshka Networks as a method to predict 3D geometry using nested shape layers.

ABSTRACT

In this paper, we develop novel, efficient 2D encodings for 3D geometry, which enable reconstructing full 3D shapes from a single image at high resolution. The key idea is to pose 3D shape reconstruction as a 2D prediction problem. To that end, we first develop a simple baseline network that predicts entire voxel tubes at each pixel of a reference view. By leveraging well-proven architectures for 2D pixel-prediction tasks, we attain state-of-the-art results, clearly outperforming purely voxel-based approaches. We scale this baseline to higher resolutions by proposing a memory-efficient shape encoding, which recursively decomposes a 3D shape into nested shape layers, similar to the pieces of a Matryoshka doll. This allows reconstructing highly detailed shapes with complex topology, as demonstrated in extensive experiments; we clearly outperform previous octree-based approaches despite having a much simpler architecture using standard network components. Our Matryoshka networks further enable reconstructing shapes from IDs or shape similarity, as well as shape sampling.

Motivation & Objective

• Introduce a novel network architecture called Matryoshka Networks for 3D geometry prediction.
• Propose the concept of nested shape layers as an efficient representation for 3D shapes.
• Demonstrate how nested layers enable improved reasoning about 3D structure.

Proposed method

• Propose a neural architecture that uses nested shape layers to represent 3D geometry.
• Leverage the nested representation to infer detailed 3D structure from input data.
• Outline training procedures to optimize the nested layer model for geometry prediction.

Experimental results

Research questions

• RQ1 What are the benefits of representing 3D geometry with nested shape layers compared to traditional representations?
• RQ2 How do Matryoshka Networks perform in predicting 3D geometry from given inputs?
• RQ3 What are the implications of nested shape layers for efficiency and accuracy in 3D prediction?

Key findings

• Not available in the provided excerpt of the source.
• The exact quantitative results and findings are not contained in the provided text.