[Paper Review] Controlled Face Manipulation and Synthesis for Data Augmentation

Joris Kirchner, Amogh Gudi+2 more|arXiv (Cornell University)|Feb 22, 2026

Face recognition and analysis0 citations

TL;DR

The paper introduces a semantic-latent-space facial manipulation method (Diffusion Autoencoder) for data augmentation, focusing on controlling Action Units (AUs) with reduced entanglement and artifacts. It shows improved AU detector training and more diverse, identity-preserving synthesis.

ABSTRACT

Deep learning vision models excel with abundant supervision, but many applications face label scarcity and class imbalance. Controllable image editing can augment scarce labeled data, yet edits often introduce artifacts and entangle non-target attributes. We study this in facial expression analysis, targeting Action Unit (AU) manipulation where annotation is costly and AU co-activation drives entanglement. We present a facial manipulation method that operates in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder). Using lightweight linear models, we reduce entanglement of semantic features via (i) dependency-aware conditioning that accounts for AU co-activation, and (ii) orthogonal projection that removes nuisance attribute directions (e.g., glasses), together with an expression neutralization step to enable absolute AU edit. We use these edits to balance AU occurrence by editing labeled faces and to diversify identities/demographics via controlled synthesis. Augmenting AU detector training with the generated data improves accuracy and yields more disentangled predictions with fewer co-activation shortcuts, outperforming alternative data-efficient training strategies and suggesting improvements similar to what would require substantially more labeled data in our learning-curve analysis. Compared to prior methods, our edits are stronger, produce fewer artifacts, and preserve identity better.

Motivation & Objective

• Address label scarcity and class imbalance in face analysis tasks.
• Develop a controllable, artifact-minimizing face manipulation method for AU editing.
• Balance AU occurrences while preserving identity and diversity in synthesized data.
• Enable absolute AU edits via an expression neutralization step.
• Demonstrate data augmentation gains for AU detectors with stronger edits than prior methods.

Proposed method

• Operate edits in the semantic latent space of a pre-trained face generator (Diffusion Autoencoder).
• Use lightweight linear models to reduce entanglement of semantic features.
• Apply dependency-aware conditioning to account for AU co-activation.
• Apply orthogonal projection to remove nuisance attribute directions (e.g., glasses).
• Include an expression neutralization step to enable absolute AU edits.
• Use edited data to balance AU occurrences and diversify identities/demographics.

Experimental results

Research questions

• RQ1 Can controllable facial edits in a diffusion-based latent space achieve stronger AU manipulation with fewer artifacts?
• RQ2 Does dependency-aware conditioning and orthogonal projection reduce AU entanglement and non-target attribute leakage?
• RQ3 Does augmented AU training with the proposed edits improve detector accuracy and disentanglement compared to baseline data-efficient strategies?
• RQ4 How well do the edits preserve identity while enabling diverse demographic and identity synthesis?

Key findings

• Edits in the semantic latent space produce stronger AU manipulation with fewer artifacts than prior methods.
• Dependency-aware conditioning and orthogonal projection reduce entanglement from AU co-activation and nuisance attributes.
• Expression neutralization enables absolute AU edits, aiding balanced AU data augmentation.
• Augmenting AU detector training with generated data improves accuracy and yields more disentangled predictions.
• The proposed method surpasses alternative data-efficient training strategies in improving AU detection performance and data diversity.