[Paper Review] Demography-based Facial Retouching Detection using Subclass Supervised Sparse Autoencoder

Aparna Bharati, Mayank Vatsa+3 more|arXiv (Cornell University)|Sep 22, 2017

Face recognition and analysis27 references3 citations

TL;DR

This paper proposes a novel subclass supervised sparse autoencoder (S³A) for demography-aware facial retouching detection, introducing the Multi-Demographic Retouched Faces (MDRF) dataset with male/female and three ethnicities (Caucasian, Chinese, Indian) across two retouching tools. The method improves detection accuracy across diverse demographics, achieving 95.9% and 94.3% for males and females respectively, outperforming prior state-of-the-art models, especially in cross-ethnicity settings.

ABSTRACT

Digital retouching of face images is becoming more widespread due to the introduction of software packages that automate the task. Several researchers have introduced algorithms to detect whether a face image is original or retouched. However, previous work on this topic has not considered whether or how accuracy of retouching detection varies with the demography of face images. In this paper, we introduce a new Multi-Demographic Retouched Faces (MDRF) dataset, which contains images belonging to two genders, male and female, and three ethnicities, Indian, Chinese, and Caucasian. Further, retouched images are created using two different retouching software packages. The second major contribution of this research is a novel semi-supervised autoencoder incorporating "subclass" information to improve classification. The proposed approach outperforms existing state-of-the-art detection algorithms for the task of generalized retouching detection. Experiments conducted with multiple combinations of ethnicities show that accuracy of retouching detection can vary greatly based on the demographics of the training and testing images.

Motivation & Objective

• To investigate how demographic factors such as gender and ethnicity affect the performance of facial retouching detection algorithms.
• To address the gap in existing methods that do not account for demographic variability in retouching detection.
• To develop a generalized detection framework that maintains high accuracy across diverse ethnic and gender groups.
• To introduce a new benchmark dataset, MDRF, with controlled retouching across multiple demographics and tools.
• To improve model generalization through a semi-supervised autoencoder incorporating subclass-level supervision.

Proposed method

• Proposes a Subclass Supervised Sparse Autoencoder (S³A) that integrates subclass information into sparse autoencoder training to enhance feature learning for retouching detection.
• Uses the MDRF dataset, which includes 325 subjects across three ethnicities (Caucasian, Chinese, Indian) and two genders, with images retouched using two distinct software tools (BeautyPlus and PortraitPro).
• Applies a two-stage learning process: pre-training the sparse autoencoder on unlabeled data, followed by fine-tuning with subclass-level supervision to distinguish original vs. retouched images.
• Employs VGG-Face features as input to the S³A framework, combining deep representation learning with sparse coding for improved discriminability.
• Introduces a novel loss function that encourages sparsity while preserving discriminative power by leveraging gender and ethnicity as subclass labels.
• Evaluates performance using cross-ethnicity and cross-gender training/testing protocols to assess generalization across demographic groups.

Experimental results

Research questions

• RQ1 How does the demographic composition of training and testing data affect the accuracy of facial retouching detection?
• RQ2 Can a semi-supervised autoencoder incorporating subclass information (e.g., gender and ethnicity) improve detection performance across diverse populations?
• RQ3 How does the proposed S³A model compare to existing state-of-the-art methods in terms of accuracy and generalization across different ethnicities and genders?
• RQ4 What are the relative detection difficulties across different ethnic groups (Caucasian, Chinese, Indian) when using the same retouching tools?
• RQ5 To what extent does the choice of retouching tool (e.g., BeautyPlus vs. PortraitPro) influence detection performance across demographics?

Key findings

• The proposed S³A model achieves 95.9% detection accuracy on male subjects and 94.3% on females, significantly outperforming Bharati et al. [13] (90.3% and 85.9%) and VGG+SVM (79.3% and 73.8%).
• In cross-ethnicity evaluations, the S³A model achieves 94.3% accuracy on Caucasian, 91.9% on Chinese, and 92.2% on Indian test sets when trained on Caucasian data, demonstrating strong generalization.
• The model shows the lowest standard deviation (±1.1% to ±2.3%) across all cross-ethnicity scenarios, indicating high robustness and consistency.
• Retouching is most difficult to detect on Indian faces (92.2% accuracy when trained on Indian data), while Chinese faces are the easiest to detect (97.5% accuracy when trained on Chinese data).
• The S³A framework improves detection performance across all demographic combinations, with the largest gains observed in cross-ethnicity settings, confirming its effectiveness in generalized detection.
• The study reveals that retouching from Tool 2 (PortraitPro) is slightly more challenging to detect than from Tool 1 (BeautyPlus), with detection accuracy dropping by 1.4–3.0% across demographics.