[Paper Review] BodyNet: Volumetric Inference of 3D Human Body Shapes

Gül Varol, Duygu Ceylan+5 more|arXiv (Cornell University)|Apr 13, 2018

Human Pose and Action RecognitionComputer Science57 references62 citations

TL;DR

BodyNet learns to infer full 3D volumetric human body shape from a single image using a voxel occupancy representation, multi-view projection losses, and intermediate supervision, achieving state-of-the-art SMPL fits on SURREAL and Unite the People and enabling 3D body-part segmentation.

ABSTRACT

Human shape estimation is an important task for video editing, animation and fashion industry. Predicting 3D human body shape from natural images, however, is highly challenging due to factors such as variation in human bodies, clothing and viewpoint. Prior methods addressing this problem typically attempt to fit parametric body models with certain priors on pose and shape. In this work we argue for an alternative representation and propose BodyNet, a neural network for direct inference of volumetric body shape from a single image. BodyNet is an end-to-end trainable network that benefits from (i) a volumetric 3D loss, (ii) a multi-view re-projection loss, and (iii) intermediate supervision of 2D pose, 2D body part segmentation, and 3D pose. Each of them results in performance improvement as demonstrated by our experiments. To evaluate the method, we fit the SMPL model to our network output and show state-of-the-art results on the SURREAL and Unite the People datasets, outperforming recent approaches. Besides achieving state-of-the-art performance, our method also enables volumetric body-part segmentation.

Motivation & Objective

• Address single-view 3D human body shape estimation.
• Introduce a volumetric voxel-based representation for 3D human shapes.
• Develop an end-to-end network with multi-view re-projection and intermediate supervision (2D pose, 2D segmentation, 3D pose).
• Achieve state-of-the-art SMPL fitting results on SURREAL and Unite the People datasets.
• Enable volumetric 3D body-part segmentation alongside shape prediction.

Proposed method

• Predict 3D body shape as a voxel occupancy grid centered at the root joint with 128x128x128 resolution.
• Train with a voxel-wise binary cross-entropy loss and extend to 3D body-part segmentation with a multi-class cross-entropy loss.
• Use multi-view re-projection losses by projecting the voxel grid to front and side views to emphasize boundary voxels (FV and SV losses).
• Employ a multi-task learning setup with intermediate supervision for 2D pose, 2D body part segmentation, and 3D pose to aid 3D shape estimation.
• Use a staged training curriculum: train 2D tasks, then 3D pose, then 3D shape with re-projection losses, followed by end-to-end fine-tuning with a combined loss.
• For evaluation, fit the SMPL model to voxel outputs by isosurface extraction and optimization of SMPL parameters via Chamfer distance plus a joint alignment term.

Experimental results

Research questions

• RQ1 Can a voxel-based volumetric representation yield accurate single-view 3D human body shapes from RGB images?
• RQ2 Does incorporating multi-view re-projection losses improve the quality of volumetric body reconstructions, especially for limbs?
• RQ3 Do intermediate supervisory signals (2D pose, 2D segmentation, 3D pose) improve final 3D shape estimation and SMPL fitting?
• RQ4 How does BodyNet perform on large-scale datasets like SURREAL and Unite the People (UP) compared to prior methods?
• RQ5 Can the volumetric network also produce 3D body-part segmentation without separate optimization?

Key findings

• BodyNet achieves state-of-the-art SMPL fits on SURREAL and Unite the People datasets when trained end-to-end with multi-view re-projection and auxiliary tasks.
• Using multi-view FV and SV silhouette projections improves SMPL surface error and voxel IOU over ablated variants.
• End-to-end multi-task training with intermediate supervision regularizes all subnetworks and yields better 3D shape estimation than training components independently.
• The approach enables 3D body-part segmentation and achieves 0.28s (foreground voxels) and 0.58s (body-part voxels) per image on modern GPUs.
• A beta-parameter regression network underperforms compared to voxel-based BodyNet, highlighting the advantage of learning deviations from the average SMPL shape via volumetric prediction.
• Fitting SMPL to voxel outputs yields robust alignment to the 3D pose and improves 3D metrics over competitors relying solely on 2D cues.