[Paper Review] MetaSDF: Meta-learning Signed Distance Functions

Vincent Sitzmann, Eric R. Chan+3 more|arXiv (Cornell University)|Jun 17, 2020

3D Shape Modeling and Analysis4 references91 citations

TL;DR

MetaSDF treats learning priors over neural implicit SDFs as gradient-based meta-learning, enabling fast test-time adaptation to new shapes without grid-based encoders or latent-space auto-decoders.

ABSTRACT

Neural implicit shape representations are an emerging paradigm that offers many potential benefits over conventional discrete representations, including memory efficiency at a high spatial resolution. Generalizing across shapes with such neural implicit representations amounts to learning priors over the respective function space and enables geometry reconstruction from partial or noisy observations. Existing generalization methods rely on conditioning a neural network on a low-dimensional latent code that is either regressed by an encoder or jointly optimized in the auto-decoder framework. Here, we formalize learning of a shape space as a meta-learning problem and leverage gradient-based meta-learning algorithms to solve this task. We demonstrate that this approach performs on par with auto-decoder based approaches while being an order of magnitude faster at test-time inference. We further demonstrate that the proposed gradient-based method outperforms encoder-decoder based methods that leverage pooling-based set encoders.

Motivation & Objective

• Formalize learning of a shape space as a meta-learning problem for neural implicit SDFs.
• Achieve fast test-time adaptation to new shapes from partial observations.
• Avoid reliance on regular grids and low-dimensional latent spaces while improving generalization.

Proposed method

• Represent each shape i by a neural SDF Phi_i to approximate its SDF.
• Parameterize Phi_i via conditioning of a meta-network with inner-loop specialization.
• Use gradient-based meta-learning (MAML-style) where inner-loop updates phi_i^j adapt to context X_i^{train} and outer-loop updates theta to minimize X_i^{test}.
• Initialize phi_i^0 = theta and perform k inner updates: phi_i^{j+1} = phi_i^j - alpha ⊙ ∇_{phi_i^j} L_train, with L_train computed from observed (x,s).
• Backpropagate outer-loop loss L_test through the inner-loop to update theta and per-parameter learning rates alpha.
• Advantage over auto-decoders: faster test-time inference and no strict latent-space assumption; outperforms pooling-based set encoders.

Experimental results

Research questions

• RQ1 Can gradient-based meta-learning provide a fast, flexible specialization of neural implicit SDFs from limited context observations?
• RQ2 How does MetaSDF compare to encoder-based, hypernetwork-based, and auto-decoder approaches in 2D and 3D shape generalization tasks?
• RQ3 Does META-SDF support reconstruction from zero-level-set points without grid-based inputs or fixed latent codes?
• RQ4 Is the learned shape space robust to out-of-distribution shapes and transformations?
• RQ5 What are the inference speed and reconstruction quality trade-offs relative to existing methods?

Key findings

• MetaSDF achieves performance on par with auto-decoders while being an order of magnitude faster at inference time.
• MetaSDF outperforms pooling-based set encoders in both 2D and 3D experiments and does not require a low-dimensional latent space.
• In MNIST-based 2D SDFs, MetaSDF reaches lower 1-norm errors than alternatives, particularly when using zero-level-set observations.
• In ShapeNet 3D experiments, MetaSDF delivers comparable mean/median Chamfer distances to DeepSDF while reducing inference time from ~8s to ~0.4s.
• MetaSDF can reconstruct 3D shapes conditioned on zero-level-set points and generalizes better to out-of-distribution shapes, with lower errors in several out-of-distribution tests.