[Paper Review] Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild

Chung-Yi Weng, Brian Curless+1 more|arXiv (Cornell University)|Dec 23, 2020

Human Pose and Action RecognitionComputer Science41 references22 citations

TL;DR

Vid2Actor proposes a self-supervised deep learning method to reconstruct a free-viewpoint, animatable 3D human volume from a single unstructured video of a person in the wild. By learning a canonical RGBα volume and pose-dependent motion weights, it enables high-quality novel pose and view synthesis without 3D supervision, achieving motion retargeting and bullet-time rendering on diverse real-world videos with minimal artifacts.

ABSTRACT

Given an "in-the-wild" video of a person, we reconstruct an animatable model of the person in the video. The output model can be rendered in any body pose to any camera view, via the learned controls, without explicit 3D mesh reconstruction. At the core of our method is a volumetric 3D human representation reconstructed with a deep network trained on input video, enabling novel pose/view synthesis. Our method is an advance over GAN-based image-to-image translation since it allows image synthesis for any pose and camera via the internal 3D representation, while at the same time it does not require a pre-rigged model or ground truth meshes for training, as in mesh-based learning. Experiments validate the design choices and yield results on synthetic data and on real videos of diverse people performing unconstrained activities (e.g. dancing or playing tennis). Finally, we demonstrate motion re-targeting and bullet-time rendering with the learned models.

Motivation & Objective

• To enable animatable, free-viewpoint 3D human synthesis from unconstrained, in-the-wild videos without requiring 3D supervision or pre-rigged models.
• To address the challenge of synthesizing novel poses and views from a single video with limited, diverse, and uncalibrated data.
• To develop a self-supervised framework that generalizes across diverse body poses and camera viewpoints using only image observations.
• To demonstrate practical applications such as motion retargeting and bullet-time rendering using the learned 3D representations.

Proposed method

• The method represents a person as a learned canonical RGBα volume and a set of pose-dependent volumetric motion weights, enabling re-projection to novel poses and views.
• A deep neural network is trained end-to-end on video frames to predict the canonical volume and motion weights, using a combination of L1, perceptual, and adversarial losses.
• The framework leverages volume warping to reconstruct high-resolution 3D geometry from low-resolution input, adapting ideas from prior volume-based NVS methods.
• The model is trained in a self-supervised manner using only video frames, avoiding the need for ground-truth 3D meshes, SMPL parameters, or multi-view calibration.
• Pose and view control are achieved by applying learned motion weights to the canonical volume, enabling explicit control over body pose and camera angle.
• The system uses 3D human pose estimation as input for motion retargeting and bullet-time rendering, with camera paths rotated around the subject.

Experimental results

Research questions

• RQ1 Can a 3D human representation be learned from a single unstructured video without 3D supervision or pre-rigged models?
• RQ2 How well can a self-supervised network generalize to novel poses and views not seen during training?
• RQ3 Can the learned volumetric representation support high-quality motion retargeting and bullet-time rendering without explicit 3D reconstruction?
• RQ4 What loss functions are most effective for preserving appearance and detail in free-viewpoint synthesis from unconstrained videos?
• RQ5 How does the diversity of poses and views in the input video affect the quality of the reconstructed 3D model?

Key findings

• The method achieves high-quality novel view and pose synthesis on diverse real-world videos, including dancing and tennis-playing sequences, even with as few as 300 frames.
• Perceptual losses significantly improve rendering quality, especially for facial and body details, as shown in qualitative ablation studies.
• Motion retargeting works effectively by applying mocap data to the learned model, enabling realistic animation in arbitrary poses.
• Bullet-time rendering is successfully achieved by rotating the camera around the subject using predicted 3D poses, outperforming PIFu and PIFuHD on non-frontal and complex poses.
• The model generalizes well to unseen configurations, with results improving as input video length and pose diversity increase.
• Despite limitations in handling pose-dependent appearance and specular effects, the reconstructions remain plausible and visually convincing across a wide range of views and poses.