[論文レビュー] MarrNet: 3D Shape Reconstruction via 2.5D Sketches

Jiajun Wu, Yifan Wang+4 more|arXiv (Cornell University)|Nov 8, 2017

3D Shape Modeling and Analysis参考文献 3被引用数 234

ひとこと要約

MarrNet は、単一の画像から 3D オブジェクト形状を再構成する。まず 2.5D スケッチ（深度、法線、シルエット）を推定し、それらのスケッチから differentiable reprojection consistency loss を用いて 3D ボクセル形状を回復します。

ABSTRACT

3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.

研究の動機と目的

• Motivate 3D reconstruction from a single image under strong domain transfer challenges.
• Propose a two-step, end-to-end trainable pipeline that separates 2.5D sketch estimation from full 3D shape reconstruction.
• Leverage differentiable reprojection constraints to align 3D shape with 2.5D sketches and enable self-supervised finetuning on real images.
• Demonstrate improved 3D reconstruction performance on synthetic ShapeNet data and real-world PASCAL 3D+ and IKEA datasets.
• Show that 2.5D sketches improve transferability and shape prior preservation during fine-tuning.

提案手法

• Propose MarrNet with three components: 2.5D sketch estimator (depth, normal, silhouette), 3D shape estimator (voxel-based), and a reprojection consistency loss.
• Use an encoder-decoder for 2.5D sketch estimation; encoder is ResNet-18; outputs depth, normal, silhouette at 256x256.
• Structure the 3D shape estimator as an encoder-decoder that maps 2.5D sketches to a 128x128x128 voxel grid, following TL network and 3D-VAE-GAN design cues.
• Introduce a differentiable reprojection loss that enforces consistency between the voxelized 3D shape and the estimated depth and normal maps under an orthographic projection.
• Training follows a two-step paradigm: pre-train on synthetic ShapeNet data for both 2.5D sketches (L2 loss) and 3D voxels (cross-entropy); then fine-tune on real images using the reprojection consistency loss while fixing the 3D decoder to preserve shape priors.
• Optionally, during testing, enable self-supervised fine-tuning on a single image (up to 40 iterations, ~10s).

実験結果

リサーチクエスチョン

• RQ1 Can a two-step scheme using 2.5D sketches improve single-image 3D reconstruction compared to direct RGB-to-voxel approaches?
• RQ2 Does learning 2.5D sketches transfer more readily from synthetic to real data than full 3D supervision?
• RQ3 Can differentiable 2D-3D reprojection constraints enable end-to-end finetuning on real images without annotations?
• RQ4 To what extent does fixing the 3D decoder during fine-tuning preserve learned shape priors and improve realism?
• RQ5 How does MarrNet perform on synthetic ShapeNet data and real datasets like PASCAL 3D+ and IKEA in qualitative and quantitative terms?

主な発見

• MarrNet achieves higher IoU on ShapeNet chairs than a direct RGB-to-3D baseline (IoU 0.57 vs 0.52).
• On Pascal 3D+ chairs, MarrNet outperforms the state-of-the-art DRC in user studies (45% preference? Note: use exact from text: 74% of users preferred MarrNet over DRC; 42% over ground truth).
• Fine-tuning with the decoder fixed during real-data adaptation preserves shape priors and yields more detailed 3D reconstructions than unconstrained fine-tuning.
• MarrNet better handles 3D shape reconstruction on real images (PASCAL 3D+, IKEA) and supports multiple object categories with consistent improvements in qualitative results.
• Human studies show MarrNet is preferred over DRC (74% of comparisons) and over some baseline configurations in various datasets.