[Paper Review] Human Body Measurement Estimation with Adversarial Augmentation

Nataniel Ruiz, Míriam Bellver+5 more|arXiv (Cornell University)|Oct 11, 2022

Infrared Thermography in Medicine4 citations

TL;DR

This paper proposes BMnet, a deep convolutional network that estimates 14 anthropometric body measurements from two silhouette images and height/weight. It introduces a differentiable adversarial body simulator (ABS) that generates challenging, rare body shapes (e.g., extreme BMI) to improve model robustness and accuracy, achieving up to 10% accuracy gain on real data and state-of-the-art performance, outperforming prior methods by up to 3x.

ABSTRACT

We present a Body Measurement network (BMnet) for estimating 3D anthropomorphic measurements of the human body shape from silhouette images. Training of BMnet is performed on data from real human subjects, and augmented with a novel adversarial body simulator (ABS) that finds and synthesizes challenging body shapes. ABS is based on the skinned multiperson linear (SMPL) body model, and aims to maximize BMnet measurement prediction error with respect to latent SMPL shape parameters. ABS is fully differentiable with respect to these parameters, and trained end-to-end via backpropagation with BMnet in the loop. Experiments show that ABS effectively discovers adversarial examples, such as bodies with extreme body mass indices (BMI), consistent with the rarity of extreme-BMI bodies in BMnet's training set. Thus ABS is able to reveal gaps in training data and potential failures in predicting under-represented body shapes. Results show that training BMnet with ABS improves measurement prediction accuracy on real bodies by up to 10%, when compared to no augmentation or random body shape sampling. Furthermore, our method significantly outperforms SOTA measurement estimation methods by as much as 3x. Finally, we release BodyM, the first challenging, large-scale dataset of photo silhouettes and body measurements of real human subjects, to further promote research in this area. Project website: https://adversarialbodysim.github.io

Motivation & Objective

• Address the lack of accurate, scalable methods for estimating physical body measurements from images, especially when metric accuracy is required.
• Overcome limitations in existing 3D reconstruction methods that fail to ensure metric fidelity and require complex post-processing to extract measurements.
• Mitigate data scarcity issues in training by generating synthetic, challenging body shapes that expose model weaknesses.
• Improve generalization and robustness of measurement estimation models, particularly for under-represented body types such as high-BMI individuals.
• Provide a new benchmark dataset to advance research in body measurement estimation from silhouettes.

Proposed method

• BMnet is a deep CNN that regresses 14 physical body measurements from two silhouettes, with height and weight as additional inputs to resolve scale ambiguity.
• Adversarial Body Simulator (ABS) is a differentiable, end-to-end differentiable framework that optimizes SMPL shape parameters to maximize BMnet's prediction error under fixed pose.
• ABS operates in a closed loop with BMnet, using backpropagation to generate adversarial examples that expose data distribution gaps and model vulnerabilities.
• The adversarial optimization is performed via gradient ascent on shape parameters, initialized from real BodyM subjects, using Adam with a learning rate of 0.1 over 5 steps.
• Training data is augmented with adversarial shapes from ABS, significantly improving BMnet’s performance on real-world test data without additional real data collection.
• The method leverages the SMPL body model as a differentiable 3D mesh generator, enabling differentiable rendering of silhouettes and measurement extraction via a differentiable function g.

(a)

Experimental results

Research questions

• RQ1 Can adversarial shape generation within the SMPL space effectively uncover and mitigate data distribution gaps in body measurement estimation models?
• RQ2 To what extent does adversarial augmentation improve measurement prediction accuracy on real human subjects compared to random or no augmentation?
• RQ3 How does the proposed method compare to state-of-the-art approaches in terms of mean and per-measurement error on real and simulated data?
• RQ4 Can the model generalize to under-represented body types, such as those with extreme BMI, when trained with adversarial data?
• RQ5 Does the proposed method outperform mesh reconstruction-based approaches when measuring the same physical metrics directly from predicted meshes?

Key findings

• Adversarial augmentation improved BMnet’s measurement prediction accuracy on real human bodies by up to 10% compared to training without augmentation or with random sampling.
• The method achieved state-of-the-art performance, outperforming prior SOTA methods by up to 3x in error reduction on the simulated test set.
• On the real BodyFit dataset, BMnet reduced average error by over 35% compared to recent HMR-based methods, even when height and weight were not provided.
• Adversarial augmentation significantly reduced error across all BMI categories, with consistent improvements on individual measurements, especially in the reduced-data regime (1K samples).
• The method demonstrated robustness and generalization, with notable performance gains on rare and extreme body shapes, confirming that ABS effectively discovers model failure modes.
• The release of BodyM, a large-scale dataset of 2,505 real subjects with silhouettes and 14 physical measurements, provides a new benchmark for future research in this domain.

(b)