[Paper Review] Multi-body Non-rigid Structure-from-Motion

Suryansh Kumar, Yuchao Dai+1 more|arXiv (Cornell University)|Jul 15, 2016

Advanced Vision and Imaging21 references3 citations

TL;DR

This paper proposes the first framework for multi-body non-rigid structure-from-motion (NRSFM), jointly reconstructing 3D shapes and segmenting multiple non-rigid objects from 2D feature tracks. By modeling each object's motion as lying in an affine subspace and enforcing self-expressiveness via sparse coding, the method uses ADMM to solve the non-convex optimization, achieving superior 3D reconstruction and segmentation on synthetic and real sequences compared to two-stage approaches.

ABSTRACT

Conventional structure-from-motion (SFM) research is primarily concerned with the 3D reconstruction of a single, rigidly moving object seen by a static camera, or a static and rigid scene observed by a moving camera --in both cases there are only one relative rigid motion involved. Recent progress have extended SFM to the areas of {multi-body SFM} (where there are {multiple rigid} relative motions in the scene), as well as {non-rigid SFM} (where there is a single non-rigid, deformable object or scene). Along this line of thinking, there is apparently a missing gap of "multi-body non-rigid SFM", in which the task would be to jointly reconstruct and segment multiple 3D structures of the multiple, non-rigid objects or deformable scenes from images. Such a multi-body non-rigid scenario is common in reality (e.g. two persons shaking hands, multi-person social event), and how to solve it represents a natural {next-step} in SFM research. By leveraging recent results of subspace clustering, this paper proposes, for the first time, an effective framework for multi-body NRSFM, which simultaneously reconstructs and segments each 3D trajectory into their respective low-dimensional subspace. Under our formulation, 3D trajectories for each non-rigid structure can be well approximated with a sparse affine combination of other 3D trajectories from the same structure (self-expressiveness). We solve the resultant optimization with the alternating direction method of multipliers (ADMM). We demonstrate the efficacy of the proposed framework through extensive experiments on both synthetic and real data sequences. Our method clearly outperforms other alternative methods, such as first clustering the 2D feature tracks to groups and then doing non-rigid reconstruction in each group or first conducting 3D reconstruction by using single subspace assumption and then clustering the 3D trajectories into groups.

Motivation & Objective

• Address the missing gap in SFM research: multi-body non-rigid structure-from-motion, where multiple independently deforming objects are present.
• Simultaneously solve 3D reconstruction and motion segmentation for non-rigid objects, rather than using sequential two-stage methods.
• Leverage subspace clustering principles to model each non-rigid object’s 3D trajectory as lying in a low-dimensional affine subspace.
• Develop an efficient optimization framework that jointly estimates 3D structure and segmentation using the alternating direction method of multipliers (ADMM).
• Demonstrate superior performance on complex real-world and synthetic sequences where multiple non-rigid objects interact.

Proposed method

• Model each non-rigid object’s 3D trajectory as lying in an affine subspace, enabling low-rank representation of motion.
• Enforce self-expressiveness: each 3D trajectory is represented as a sparse affine combination of others from the same structure.
• Formulate the problem as a non-convex optimization with nuclear norm and sparsity regularization, solved via ADMM.
• Use the affinity matrix from sparse coding to guide segmentation, ensuring trajectories from the same object are grouped together.
• Integrate neighboring constraints in dense sequences to improve reconstruction fidelity.
• Apply the method end-to-end on 2D feature tracks, directly outputting 3D reconstructions and segmentations.

Experimental results

Research questions

• RQ1 Can we jointly reconstruct and segment multiple non-rigid objects from a single monocular image sequence without assuming a single rigid or non-rigid motion?
• RQ2 Does modeling non-rigid motion as self-expressive affine subspaces improve 3D reconstruction and segmentation accuracy compared to two-stage methods?
• RQ3 Can the proposed ADMM-based optimization effectively handle the non-convexity of multi-body NRSFM while maintaining computational efficiency?
• RQ4 How does the method perform on complex real-world sequences with interacting non-rigid objects, such as people shaking hands or multiple gestures?
• RQ5 Can the method scale to dense feature tracks and longer video sequences without significant performance degradation?

Key findings

• On the UMPM dataset, the proposed method achieved a 3D reconstruction error of 0.1992 on p2_free_2, 0.1348 on p3_ball_1, and 0.0821 on p4_meet_12, with zero motion segmentation error (eMS = 0.0) on all sequences except p4_meet_12, where eMS = 0.0135.
• On synthetic face sequences, the method achieved a 3D reconstruction error of 0.0380 under CD constraints, outperforming baseline methods in both accuracy and robustness.
• The method achieved perfect motion segmentation (eMS = 0.0) on four of the five UMPM sequences, indicating strong clustering capability.
• Visual results on the Face+Heart sequence showed accurate segmentation with distinct color labels for each object, confirming correct grouping of non-rigid feature tracks.
• The affinity matrix from the proposed method better captured true object relationships than SSC, as shown in Figure 13, with clearer block structures indicating correct segmentation.
• The method outperformed two-stage baselines—first clustering 2D tracks then reconstructing, or first reconstructing with single subspace then clustering—by jointly optimizing both tasks in 3D space.