[Paper Review] MARLIN: Masked Autoencoder for facial video Representation LearnINg
MARLIN proposes a self-supervised masked autoencoder for learning universal facial video representations from non-annotated web-crawled videos. By using a facial part-guided tube masking strategy (Fasking) to reconstruct spatio-temporal details from densely masked facial regions, MARLIN achieves state-of-the-art performance gains across multiple downstream tasks, including 1.13% higher accuracy in Facial Attribute Recognition and 29.36% improvement in Frechet Inception Distance for Lip Synchronization.
This paper proposes a self-supervised approach to learn universal facial representations from videos, that can transfer across a variety of facial analysis tasks such as Facial Attribute Recognition (FAR), Facial Expression Recognition (FER), DeepFake Detection (DFD), and Lip Synchronization (LS). Our proposed framework, named MARLIN, is a facial video masked autoencoder, that learns highly robust and generic facial embeddings from abundantly available non-annotated web crawled facial videos. As a challenging auxiliary task, MARLIN reconstructs the spatio-temporal details of the face from the densely masked facial regions which mainly include eyes, nose, mouth, lips, and skin to capture local and global aspects that in turn help in encoding generic and transferable features. Through a variety of experiments on diverse downstream tasks, we demonstrate MARLIN to be an excellent facial video encoder as well as feature extractor, that performs consistently well across a variety of downstream tasks including FAR (1.13% gain over supervised benchmark), FER (2.64% gain over unsupervised benchmark), DFD (1.86% gain over unsupervised benchmark), LS (29.36% gain for Frechet Inception Distance), and even in low data regime. Our code and models are available at https://github.com/ControlNet/MARLIN .
Motivation & Objective
- To develop a universal, task-agnostic facial video encoder that generalizes across diverse facial analysis tasks without requiring task-specific annotations.
- To address the data scarcity and annotation cost limitations in facial analysis by leveraging abundant non-annotated facial videos for self-supervised pre-training.
- To improve feature transferability by formulating a challenging reconstruction task that captures both local (e.g., eyes, mouth) and global facial dynamics.
- To enhance representation quality through adversarial training and a novel facial region-guided masking strategy (Fasking) that outperforms standard masking methods.
- To enable effective few-shot and low-data regime performance by learning robust, generic facial features from large-scale uncurated video data.
Proposed method
- Proposes a masked autoencoder framework (MARLIN) that reconstructs spatio-temporal facial video frames from densely masked facial regions using a ViT-based encoder and decoder.
- Introduces a novel facial-guided tube masking strategy (Fasking) that dynamically tracks facial landmarks using FaceXZoo and masks key facial parts (eyes, nose, mouth, lips, skin) to encourage learning of local and global facial structure.
- Employs a masking ratio of approximately 90% to balance reconstruction difficulty and feature quality, validated empirically on CMU-MOSEI and FF++ datasets.
- Incorporates adversarial training on the latent representation to improve reconstruction fidelity and enhance the quality of learned features.
- Uses a ViT-B backbone as the default encoder, with ablation studies comparing ViT-S, ViT-B, and ViT-L for performance and scalability.
- Pre-trains on the YouTube Faces dataset using the masked autoencoding objective, followed by fine-tuning on downstream tasks with minimal labeled data.
Experimental results
Research questions
- RQ1Can a self-supervised masked autoencoder learn generic and transferable facial representations from uncurated, non-annotated facial videos?
- RQ2Does a facial part-guided masking strategy (Fasking) lead to better feature learning than random, frame-level, or standard tube masking in facial video representation learning?
- RQ3How does adversarial training improve the quality of reconstructed facial features and downstream performance in facial analysis tasks?
- RQ4To what extent can MARLIN generalize across diverse facial analysis tasks, including those with limited labeled data?
- RQ5What is the optimal masking ratio for balancing reconstruction difficulty and feature quality in facial video masked autoencoding?
Key findings
- MARLIN achieves a 1.13% absolute accuracy gain over the supervised benchmark in Facial Attribute Recognition (FAR), demonstrating strong transferability even with limited labeled data.
- In Facial Expression Recognition (FER), MARLIN improves performance by 2.64% over the unsupervised benchmark, highlighting its effectiveness in learning emotion-sensitive representations.
- For DeepFake Detection (DFD), MARLIN achieves a 1.86% gain over the unsupervised baseline, indicating robustness to subtle facial manipulation patterns.
- In Lip Synchronization (LS), MARLIN improves the Frechet Inception Distance (FID) by 29.36% compared to baseline methods, showing superior generation quality of spatio-temporal facial motion.
- The ablation study confirms that the Fasking masking strategy outperforms random, frame-level, and standard tube masking, with the highest performance on all downstream tasks.
- The optimal masking ratio is empirically found to be approximately 90%, as both lower and higher ratios degrade performance due to either too much visible information or excessive reconstruction difficulty.
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.