[Paper Review] Dense Non-Rigid Structure from Motion: A Manifold Viewpoint

Suryansh Kumar, Luc Van Gool+4 more|arXiv (Cornell University)|Jun 15, 2020

Advanced Vision and Imaging58 references4 citations

TL;DR

This paper proposes a manifold-based approach to dense non-rigid structure from motion by modeling local linear subspaces using Grassmann manifolds and global low-rank shape representations, enabling accurate, scalable, and robust 3D reconstruction of complex deformations. The method achieves state-of-the-art performance on benchmark datasets by jointly optimizing reconstruction and clustering through spatial-temporal manifold fusion.

ABSTRACT

Non-Rigid Structure-from-Motion (NRSfM) problem aims to recover 3D geometry of a deforming object from its 2D feature correspondences across multiple frames. Classical approaches to this problem assume a small number of feature points and, ignore the local non-linearities of the shape deformation, and therefore, struggles to reliably model non-linear deformations. Furthermore, available dense NRSfM algorithms are often hurdled by scalability, computations, noisy measurements and, restricted to model just global deformation. In this paper, we propose algorithms that can overcome these limitations with the previous methods and, at the same time, can recover a reliable dense 3D structure of a non-rigid object with higher accuracy. Assuming that a deforming shape is composed of a union of local linear subspace and, span a global low-rank space over multiple frames enables us to efficiently model complex non-rigid deformations. To that end, each local linear subspace is represented using Grassmannians and, the global 3D shape across multiple frames is represented using a low-rank representation. We show that our approach significantly improves accuracy, scalability, and robustness against noise. Also, our representation naturally allows for simultaneous reconstruction and clustering framework which in general is observed to be more suitable for NRSfM problems. Our method currently achieves leading performance on the standard benchmark datasets.

Motivation & Objective

• Address the limitations of existing sparse and dense NRSfM methods in handling complex, non-linear deformations and scalability.
• Overcome the failure of global low-rank assumptions in modeling local shape deformations common in dense surface motion.
• Develop a scalable, noise-robust framework that integrates both local and global deformation constraints for dense 3D reconstruction.
• Enable simultaneous reconstruction and clustering through a unified manifold representation.
• Achieve leading performance on standard NRSfM benchmarks with improved robustness to noise and computational efficiency.

Proposed method

• Model local shape deformations using Grassmann manifolds to represent local linear subspaces in both space and time.
• Represent the global 3D shape across frames using a low-rank matrix factorization, assuming the shape lies in a low-dimensional subspace.
• Use dimensionality reduction on the Grassmann manifold to improve clustering and robustness to noise.
• Formulate a joint optimization that fuses spatial and temporal information across shape and trajectory manifolds.
• Apply a threshold-based dimension selection (τ = 0.97) to retain 97% of the variance in singular values for optimal subspace representation.
• Extend the framework to handle noisy inputs and unknown temporal shape information by leveraging neighboring trajectory similarity.

Experimental results

Research questions

• RQ1 Can local linear subspaces on Grassmann manifolds improve modeling of complex non-rigid deformations compared to global low-rank assumptions?
• RQ2 How can local and global deformation constraints be jointly optimized to enhance dense 3D reconstruction accuracy?
• RQ3 To what extent does dimensionality reduction on Grassmann manifolds improve robustness to noise in NRSfM?
• RQ4 Can a unified spatial-temporal manifold representation enable simultaneous reconstruction and clustering in dense NRSfM?
• RQ5 How does the method perform under varying camera motion and noisy input compared to existing state-of-the-art approaches?

Key findings

• The proposed method achieves state-of-the-art performance on standard benchmark datasets for dense non-rigid structure from motion.
• Using τ = 0.97 for dimension selection consistently yields optimal reconstruction accuracy across diverse datasets.
• The method significantly improves robustness to noise by projecting manifold representations into lower-dimensional subspaces before clustering.
• The joint reconstruction and clustering framework enables better handling of complex, non-linear deformations than prior global low-rank models.
• The algorithm scales effectively to dense feature points, overcoming limitations of previous sparse NRSfM methods when applied to dense settings.
• The approach outperforms existing dense NRSfM algorithms in accuracy and computational efficiency, particularly under noisy conditions.