[Paper Review] Model-based Outdoor Performance Capture

Nadia Robertini, Dan Casas+3 more|arXiv (Cornell University)|Oct 21, 2016

Human Pose and Action Recognition48 references7 citations

TL;DR

This paper proposes a novel model-based method for high-fidelity human performance capture in uncontrolled outdoor environments using a unified implicit representation of 3D Gaussians. By jointly optimizing skeletal pose and non-rigid surface shape in two stages—first coarse pose estimation, then fine shape refinement using 'Border Gaussians'—the method achieves state-of-the-art reconstruction quality outdoors without explicit silhouette segmentation, matching indoor performance of prior methods.

ABSTRACT

We propose a new model-based method to accurately reconstruct human performances captured outdoors in a multi-camera setup. Starting from a template of the actor model, we introduce a new unified implicit representation for both, articulated skeleton tracking and nonrigid surface shape refinement. Our method fits the template to unsegmented video frames in two stages - first, the coarse skeletal pose is estimated, and subsequently non-rigid surface shape and body pose are jointly refined. Particularly for surface shape refinement we propose a new combination of 3D Gaussians designed to align the projected model with likely silhouette contours without explicit segmentation or edge detection. We obtain reconstructions of much higher quality in outdoor settings than existing methods, and show that we are on par with state-of-the-art methods on indoor scenes for which they were designed

Motivation & Objective

• To address the challenge of high-quality human performance capture in uncontrolled outdoor scenes where traditional silhouette-based methods fail due to variable lighting and dynamic backgrounds.
• To enable accurate reconstruction of both articulated body pose and non-rigid surface geometry without requiring explicit foreground segmentation or edge detection.
• To develop a unified implicit representation using 3D Gaussians that jointly models skeletal pose and surface shape for improved optimization and temporal coherence.
• To achieve reconstruction quality on par with state-of-the-art indoor methods while significantly outperforming existing approaches in outdoor settings.
• To demonstrate robustness and accuracy in real-world outdoor sequences with complex backgrounds and dynamic lighting.

Proposed method

• The method uses a two-stage fitting process: first optimizing coarse skeletal pose using volumetric 3D Gaussians, then jointly refining pose and non-rigid surface shape.
• A unified implicit representation based on 3D Gaussians models both coarse body shape and fine surface geometry, enabling differentiable and efficient optimization.
• The surface refinement stage introduces 'Border Gaussians'—a specialized combination of 3D Gaussians designed to align projected model silhouettes with likely contour regions without explicit segmentation.
• Each input image is also transformed into an implicit representation, enabling consistent comparison between observed and predicted silhouettes.
• The objective function minimizes photometric and geometric discrepancies between the projected model and input images, using differentiable rendering and implicit function optimization.
• The method operates on unsegmented video frames, avoiding reliance on precomputed silhouettes or background subtraction.

Experimental results

Research questions

• RQ1 Can a unified implicit representation of 3D Gaussians enable accurate joint optimization of skeletal pose and non-rigid surface shape in unsegmented outdoor video?
• RQ2 Can 'Border Gaussians' effectively guide surface refinement toward correct silhouette contours without explicit edge detection or segmentation?
• RQ3 How does the two-stage fitting pipeline (coarse pose then joint refinement) improve reconstruction quality in challenging outdoor scenes compared to single-stage or segmentation-dependent methods?
• RQ4 To what extent can this model-based approach achieve performance comparable to state-of-the-art indoor methods in outdoor settings with dynamic backgrounds and variable lighting?
• RQ5 Can the method maintain temporal coherence and high geometric fidelity in sequences with loose clothing and complex deformations without manual intervention?

Key findings

• The method achieves an F1 score of 0.9362 ± 0.0033 on the cathedral sequence and 0.9223 ± 0.0083 on the unicampus sequence after mesh refinement, significantly outperforming Stage-I results.
• On the skirt sequence with known color background, the method achieves an F1 score of 0.9676 ± 0.0056, comparable to the silhouette-based method of Gall et al. (0.9683 ± 0.0045).
• The use of both Surface and Border Gaussians in refinement leads to a consistent and significant improvement in silhouette overlap over Stage-I alone.
• Qualitative results show temporally coherent, detailed 3D meshes without noise or geometric artifacts, even in complex outdoor scenes with moving backgrounds.
• The method successfully reconstructs textured models by reprojecting original frames, demonstrating accurate surface geometry and appearance alignment.
• The approach achieves high-quality reconstructions in outdoor settings where prior model-based methods fail, while matching the performance of indoor-state-of-the-art methods.