[Paper Review] FLEX: Parameter-free Multi-view 3D Human Motion Reconstruction.

Brian Gordon, Sigal Raab+3 more|arXiv (Cornell University)|May 5, 2021

Human Pose and Action Recognition48 references4 citations

TL;DR

FLEX is a parameter-free, end-to-end multi-view 3D human motion reconstruction model that eliminates dependency on camera parameters by learning invariant 3D joint rotations and bone lengths across views. It uses a novel multi-view fusion layer to combine video streams and reconstructs temporally coherent, consistent 3D skeletons, outperforming state-of-the-art methods without camera parameters while matching their performance when parameters are provided.

ABSTRACT

The increasing availability of video recordings made by multiple cameras has offered new means for mitigating occlusion and depth ambiguities in pose and motion reconstruction methods. Yet, multi-view algorithms strongly depend on camera parameters, in particular, the relative positions among the cameras. Such dependency becomes a hurdle once shifting to dynamic capture in uncontrolled settings. We introduce FLEX (Free muLti-view rEconstruXion), an end-to-end parameter-free multi-view model. FLEX is parameter-free in the sense that it does not require any camera parameters, neither intrinsic nor extrinsic. Our key idea is that the 3D angles between skeletal parts, as well as bone lengths, are invariant to the camera position. Hence, learning 3D rotations and bone lengths rather than locations allows predicting common values for all camera views. Our network takes multiple video streams, learns fused deep features through a novel multi-view fusion layer, and reconstructs a single consistent skeleton with temporally coherent joint rotations. We demonstrate quantitative and qualitative results on the Human3.6M and KTH Multi-view Football II datasets. We compare our model to state-of-the-art methods that are not parameter-free and show that in the absence of camera parameters, we outperform them by a large margin while obtaining comparable results when camera parameters are available. Code, trained models, video demonstration, and additional materials will be available on our project page.

Motivation & Objective

• To address the limitation of multi-view 3D human motion reconstruction methods that rely on precise camera parameters, especially in uncontrolled, dynamic environments.
• To eliminate dependency on intrinsic and extrinsic camera parameters, enabling deployment in real-world, unconstrained settings.
• To develop a unified, end-to-end model that reconstructs consistent and temporally coherent 3D human skeletons from multiple video streams without requiring camera calibration.
• To leverage geometric invariance—specifically, 3D joint angles and bone lengths—across camera views to improve robustness and generalization.

Proposed method

• FLEX learns 3D joint rotations and bone lengths as invariant features across multiple camera views, bypassing the need for 3D joint locations.
• It employs a novel multi-view fusion layer that aggregates deep features from multiple video streams into a shared representation.
• The network is trained end-to-end to reconstruct a single, consistent 3D skeleton sequence with temporal coherence.
• The model leverages the geometric invariance of 3D angles and bone lengths under different camera perspectives to unify predictions across views.
• It uses a siamese-like architecture to ensure consistent 3D pose estimation across all camera views without explicit camera parameter supervision.
• The architecture is trained using a combination of 3D keypoint regression and temporal consistency loss to ensure smooth, realistic motion sequences.

Experimental results

Research questions

• RQ1 Can 3D human motion reconstruction be achieved without relying on camera parameters such as intrinsic or extrinsic calibration?
• RQ2 How can geometric invariance—specifically 3D joint angles and bone lengths—be exploited to unify multi-view predictions across uncalibrated cameras?
• RQ3 To what extent can a parameter-free model outperform state-of-the-art methods that require camera parameters in uncontrolled, dynamic environments?
• RQ4 Can a deep learning model achieve consistent and temporally coherent 3D motion reconstruction across multiple uncalibrated video streams?

Key findings

• FLEX outperforms state-of-the-art methods in the absence of camera parameters on both Human3.6M and KTH Multi-view Football II datasets.
• When camera parameters are provided, FLEX achieves performance comparable to the best existing parameter-dependent methods.
• The model demonstrates robustness in uncontrolled, dynamic settings due to its parameter-free design and reliance on geometric invariants.
• The use of a novel multi-view fusion layer enables effective feature aggregation across views, improving reconstruction accuracy.
• Temporal consistency loss significantly improves the smoothness and realism of the reconstructed motion sequences.
• The model generalizes well across diverse camera configurations, confirming the effectiveness of learning invariant 3D rotations and bone lengths.