[Paper Review] Live Face De-Identification in Video

Oran Gafni, Lior Wolf+1 more|arXiv (Cornell University)|Nov 19, 2019

Face recognition and analysis48 references4 citations

TL;DR

This paper proposes a novel, feed-forward encoder-decoder network for live video de-identification that preserves pose, expression, illumination, and identity-agnostic appearance while fully decorrelating identity. It achieves high-fidelity, artifact-free video modification by conditioning on a pre-trained face recognition network's features and using a novel attractor-repeller perceptual loss, enabling real-time, identity-destroying face transformation without retraining.

ABSTRACT

We propose a method for face de-identification that enables fully automatic video modification at high frame rates. The goal is to maximally decorrelate the identity, while having the perception (pose, illumination and expression) fixed. We achieve this by a novel feed-forward encoder-decoder network architecture that is conditioned on the high-level representation of a person's facial image. The network is global, in the sense that it does not need to be retrained for a given video or for a given identity, and it creates natural looking image sequences with little distortion in time.

Motivation & Objective

• To enable fully automatic, real-time video de-identification that preserves non-identity attributes like pose, expression, and lighting.
• To address the lack of video-capable de-identification methods that maintain perceptual consistency across frames.
• To generate new, natural-looking faces de novo rather than swapping with existing dataset faces.
• To ensure that state-of-the-art face recognition models fail to identify the de-identified subject.
• To develop a method robust to diverse poses, lighting, occlusions, and facial structures without retraining.

Proposed method

• A feed-forward encoder-decoder architecture that conditions on the high-level representation of a face from a pre-trained face recognition network.
• Latent space injection of the face recognition network's features to guide identity-agnostic generation without retraining.
• A novel attractor-repeller perceptual loss that distinguishes low/mid-level features (for frame consistency) from high-level features (for identity distancing).
• Simultaneous output of a face image and a segmentation mask for precise blending and reconstruction.
• Training with reconstruction loss, edge loss, adversarial loss, and a semantic data augmentation technique to preserve structural semantics.
• Hyperparameter control via λ to adjust the strength of identity distancing while maintaining perceptual fidelity.

Experimental results

Research questions

• RQ1 Can a single, non-retraining model generate high-quality, identity-destroyed faces in real-time video while preserving pose, expression, and illumination?
• RQ2 How can perceptual consistency be maintained across video frames during identity de-identification?
• RQ3 To what extent can a learned representation from a pre-trained face classifier be used to guide identity-destroying generation?
• RQ4 How does the proposed attractor-repeller perceptual loss improve identity distancing while minimizing visual artifacts?
• RQ5 Can the method generalize across diverse identities, poses, and lighting conditions without fine-tuning?

Key findings

• The method produces video sequences with minimal temporal flicker and distortion, maintaining consistent pose, expression, and illumination across frames.
• State-of-the-art face recognition models fail to identify the de-identified subjects, confirming effective identity de-identification.
• Human observers cannot identify the de-identified faces even with extended viewing time, indicating strong perceptual anonymity.
• The method outperforms prior works in preserving low- and mid-level features such as mouth expression and facial hair, which are often distorted in previous methods.
• The use of a learned mask enables high-resolution face generation with precise blending, avoiding artifacts common in full-body generation approaches.
• Ablation studies confirm that removing key components like the attractor-repeller loss or mask output leads to visible artifacts and reduced identity distancing.