[Paper Review] Monocular Real-Time Volumetric Performance Capture

Ruilong Li, Yuliang Xiu+4 more|arXiv (Cornell University)|Jul 28, 2020

Advanced Vision and Imaging84 references4 citations

TL;DR

This paper presents the first real-time monocular volumetric performance capture system that reconstructs high-fidelity, fully textured 3D humans from a single video stream without requiring multi-view setups or personalized templates. By introducing a hierarchical surface localization algorithm and mesh-free rendering, the method accelerates reconstruction by two orders of magnitude and achieves 15 fps at 256³ resolution, while an Online Hard Example Mining technique improves robustness to rare poses and clothing styles.

ABSTRACT

We present the first approach to volumetric performance capture and novel-view rendering at real-time speed from monocular video, eliminating the need for expensive multi-view systems or cumbersome pre-acquisition of a personalized template model. Our system reconstructs a fully textured 3D human from each frame by leveraging Pixel-Aligned Implicit Function (PIFu). While PIFu achieves high-resolution reconstruction in a memory-efficient manner, its computationally expensive inference prevents us from deploying such a system for real-time applications. To this end, we propose a novel hierarchical surface localization algorithm and a direct rendering method without explicitly extracting surface meshes. By culling unnecessary regions for evaluation in a coarse-to-fine manner, we successfully accelerate the reconstruction by two orders of magnitude from the baseline without compromising the quality. Furthermore, we introduce an Online Hard Example Mining (OHEM) technique that effectively suppresses failure modes due to the rare occurrence of challenging examples. We adaptively update the sampling probability of the training data based on the current reconstruction accuracy, which effectively alleviates reconstruction artifacts. Our experiments and evaluations demonstrate the robustness of our system to various challenging angles, illuminations, poses, and clothing styles. We also show that our approach compares favorably with the state-of-the-art monocular performance capture. Our proposed approach removes the need for multi-view studio settings and enables a consumer-accessible solution for volumetric capture.

Motivation & Objective

• To enable real-time, high-quality volumetric performance capture from a single consumer-grade camera, eliminating the need for multi-view systems or pre-captured templates.
• To overcome the computational bottleneck of Pixel-Aligned Implicit Function (PIFu) for real-time deployment, which traditionally takes tens of seconds per frame.
• To improve reconstruction robustness to challenging conditions such as rare poses, complex clothing, and varying lighting by addressing data imbalance in training.
• To achieve direct, efficient novel-view rendering without explicit mesh extraction, reducing rendering latency.

Proposed method

• Proposes a hierarchical surface localization algorithm that queries 3D points in a coarse-to-fine manner, drastically reducing the number of evaluations needed for surface reconstruction.
• Introduces a direct rendering technique that bypasses mesh extraction and texture mapping, enabling faster novel-view synthesis by rendering directly from the implicit function.
• Employs an Online Hard Example Mining (OHEM) strategy that adaptively reweights training samples based on reconstruction error, focusing learning on difficult cases.
• Uses a progressive sampling strategy that increases the likelihood of sampling hard examples (e.g., rare poses, complex clothing) during training to suppress artifacts.
• Leverages the PIFu framework for high-resolution, implicit 3D representation of geometry and texture, while optimizing inference speed through spatial pruning.
• Combines the hierarchical localization with implicit texture rendering to achieve end-to-end real-time performance at 256³ resolution.

Experimental results

Research questions

• RQ1 Can monocular 3D human performance capture be accelerated to real-time speeds without sacrificing reconstruction quality or requiring multi-view data?
• RQ2 How can the high computational cost of PIFu-style implicit reconstruction be reduced while preserving high-fidelity geometry and texture?
• RQ3 Can data imbalance in synthetic training data—especially for rare poses and clothing styles—be effectively mitigated to improve generalization and reduce artifacts?
• RQ4 Is it feasible to achieve real-time novel-view rendering without explicitly extracting and tessellating surface meshes?
• RQ5 Can an adaptive sampling strategy during training improve robustness to challenging input conditions without manual data curation?

Key findings

• The proposed hierarchical surface localization algorithm accelerates PIFu-based reconstruction by two orders of magnitude, reducing inference time from 30 seconds to 0.14 seconds per frame.
• By combining the accelerated reconstruction with mesh-free rendering, the system achieves 15 fps for full volumetric rendering at 256³ resolution, enabling real-time performance capture.
• The Online Hard Example Mining (OHEM) technique significantly reduces reconstruction artifacts, especially in rare poses and complex clothing, improving overall fidelity and generalization.
• Qualitative and quantitative evaluations show the method outperforms state-of-the-art monocular reconstruction approaches in accuracy and robustness, even without a pre-captured template.
• The system successfully handles topology changes and dynamic textures, enabling realistic rendering of expressive performances, unlike template-based methods.
• The approach generalizes beyond humans and can be applied to any object or topology, making it a scalable foundation for real-time 3D capture from consumer cameras.