[Paper Review] "Maximizing rigidity" revisited: a convex programming approach for generic 3D shape reconstruction from multiple perspective views

Pan Ji, Hongdong Li+2 more|arXiv (Cornell University)|Jul 17, 2017

Advanced Vision and Imaging3 citations

TL;DR

This paper proposes a convex semi-definite programming framework that unifies rigid and non-rigid 3D shape reconstruction from multiple perspective views by maximizing rigidity without explicit camera pose estimation. It formulates the problem as a convex optimization that minimizes non-rigidity, achieving state-of-the-art accuracy on rigid, non-rigid, and articulated motion datasets with global optimality guarantees.

ABSTRACT

Rigid structure-from-motion (RSfM) and non-rigid structure-from-motion (NRSfM) have long been treated in the literature as separate (different) problems. Inspired by a previous work which solved directly for 3D scene structure by factoring the relative camera poses out, we revisit the principle of "maximizing rigidity" in structure-from-motion literature, and develop a unified theory which is applicable to both rigid and non-rigid structure reconstruction in a rigidity-agnostic way. We formulate these problems as a convex semi-definite program, imposing constraints that seek to apply the principle of minimizing non-rigidity. Our results demonstrate the efficacy of the approach, with state-of-the-art accuracy on various 3D reconstruction problems.

Motivation & Objective

• To unify rigid and non-rigid structure-from-motion (SfM) into a single framework, avoiding separate treatment of the two cases.
• To overcome the limitations of traditional two-stage SfM pipelines that require explicit camera motion estimation and separate methods for rigid vs. non-rigid scenes.
• To develop a rigidity-agnostic method that automatically determines the optimal 3D structure based on observed image measurements.
• To reformulate the 'maximizing rigidity' principle using a convex rigidity measure, enabling global optimization via semi-definite programming.
• To achieve state-of-the-art reconstruction accuracy across diverse 3D motion types, including deforming, articulated, and rigid objects.

Proposed method

• The method directly reconstructs 3D structure by building viewing triangles from 2D feature correspondences, bypassing explicit camera pose estimation.
• It formulates the 3D reconstruction problem as a convex semi-definite program (SDP) that minimizes a novel convex rigidity measure to enforce structural consistency.
• The rigidity measure is derived from point-to-point distance deviations, promoting solutions that are as rigid as possible under the image constraints.
• The approach uses SDP relaxation to ensure global optimality, avoiding local minima common in non-convex formulations.
• For each view, the method solves an SDP of size (n+1)×(n+1), where n is the number of 3D points, to jointly estimate structure and enforce rigidity.
• The framework is inherently rigidity-agnostic, automatically adapting to rigid, non-rigid, or articulated scenes based on data.

Experimental results

Research questions

• RQ1 Can a unified 3D reconstruction framework be developed that handles both rigid and non-rigid scenes without prior knowledge of the scene type?
• RQ2 Can the principle of 'maximizing rigidity' be reformulated as a convex optimization problem to ensure global optimality?
• RQ3 Does a convex rigidity measure outperform non-convex alternatives in terms of reconstruction accuracy and robustness?
• RQ4 Can the method achieve state-of-the-art performance on diverse 3D reconstruction benchmarks, including articulated motion and non-rigid deformation?
• RQ5 How does the proposed method compare to existing convex formulations in terms of accuracy and computational scalability?

Key findings

• On the CMU Mocap dataset, the method achieved a mean relative 3D error of 1.11% on the point-articulated sequence and 0.45% on the axis-articulated sequence, significantly outperforming the baseline [3].
• For the Dance, Drink, Pickup, Yoga, and Stretch sequences, the proposed method showed consistently lower mean 3D reconstruction errors than the state-of-the-art baseline [3].
• On the HumanEva-I dataset, the method achieved an RMSE of 0.162 mm and a relative error of 3.02%, outperforming the baseline [3] which reported 0.200 mm and 3.73%.
• The method achieved state-of-the-art accuracy on the HumanEva-I dataset, demonstrating superior performance on non-rigid shape reconstruction.
• The framework successfully reconstructed articulated motions with low error, indicating robustness to complex motion types.
• The convex formulation guarantees global optimality, avoiding local minima that plague non-convex approaches.