[Paper Review] Can you tell where in India I am from? Comparing humans and computers on fine-grained race face classification

Harish Katti, S. P. Arun|arXiv (Cornell University)|Mar 22, 2017

Face recognition and analysis9 references3 citations

TL;DR

This study introduces a fine-grained race classification task distinguishing North vs. South Indian faces using a novel dataset of 1,647 diverse faces with human labeling and performance data from 129 subjects. Despite machines achieving comparable overall accuracy (64%), their error patterns differed systematically from humans, revealing that mouth shape is the most discriminative facial feature—confirmed by occlusion experiments showing human performance dropped most when the mouth was obscured.

ABSTRACT

Faces form the basis for a rich variety of judgments in humans, yet the underlying features remain poorly understood. Although fine-grained distinctions within a race might more strongly constrain possible facial features used by humans than in case of coarse categories such as race or gender, such fine grained distinctions are relatively less studied. Fine-grained race classification is also interesting because even humans may not be perfectly accurate on these tasks. This allows us to compare errors made by humans and machines, in contrast to standard object detection tasks where human performance is nearly perfect. We have developed a novel face database of close to 1650 diverse Indian faces labeled for fine-grained race (South vs North India) as well as for age, weight, height and gender. We then asked close to 130 human subjects who were instructed to categorize each face as belonging toa Northern or Southern state in India. We then compared human performance on this task with that of computational models trained on the ground-truth labels. Our main results are as follows: (1) Humans are highly consistent (average accuracy: 63.6%), with some faces being consistently classified with > 90% accuracy and others consistently misclassified with < 30% accuracy; (2) Models trained on ground-truth labels showed slightly worse performance (average accuracy: 62%) but showed higher accuracy (72.2%) on faces classified with > 80% accuracy by humans. This was true for models trained on simple spatial and intensity measurements extracted from faces as well as deep neural networks trained on race or gender classification; (3) Using overcomplete banks of features derived from each face part, we found that mouth shape was the single largest contributor towards fine-grained race classification, whereas distances between face parts was the strongest predictor of gender.

Motivation & Objective

• To investigate human performance on a hard, fine-grained race classification task distinguishing North and South Indian faces.
• To identify the specific facial features humans use for such fine-grained discrimination, overcoming limitations of coarse race categories.
• To compare machine learning models' performance and error patterns against human performance on the same task.
• To validate the importance of specific facial regions using behavioral occlusion experiments.
• To uncover systematic differences in representational learning between humans and machines in face recognition.

Proposed method

• Constructed a new dataset of 1,647 diverse Indian faces labeled for North vs. South Indian origin, with performance data from 129 human subjects.
• Trained linear classifiers on overcomplete sets of features extracted from individual face parts (eyes, nose, mouth, contour) to identify discriminative features.
• Conducted a controlled behavioral experiment with 24 human participants, occluding eyes, nose, or mouth in separate conditions to assess feature importance.
• Used Wilcoxon signed-rank tests to compare classification accuracy and response times across occlusion conditions.
• Evaluated multiple machine learning models on the dataset and compared their error patterns to those of humans.
• Selected 217 faces per occlusion condition (unoccluded, eye-occluded, nose-occluded, mouth-occluded), ensuring comparable baseline accuracy (~69%) across conditions.

Experimental results

Research questions

• RQ1 What facial features do humans use to distinguish fine-grained regional race differences in Indian faces?
• RQ2 How do machine learning models perform on this fine-grained classification task compared to humans?
• RQ3 Are the error patterns of machines and humans systematically different on this hard classification task?
• RQ4 Does occluding specific facial regions impair human classification accuracy, and if so, which region has the greatest impact?
• RQ5 Can computational modeling of facial features predict the relative importance of facial parts in human face recognition?

Key findings

• Humans achieved an average accuracy of 65.8% on unoccluded faces in the fine-grained North vs. South Indian face classification task.
• Machines achieved comparable overall accuracy (64%) to humans, but their error patterns across faces were qualitatively different.
• Occluding the mouth impaired human classification accuracy the most (59.8%), significantly lower than occluding eyes (63.6%) or nose (61.1%), with p < 0.0005 compared to unoccluded faces.
• Response times were significantly longer for mouth-occluded faces (p < 0.0005 vs. unoccluded), confirming greater cognitive load and reliance on mouth shape.
• Linear classifiers trained on part-specific features identified mouth shape as the most discriminative feature for race classification, supporting the behavioral findings.
• The systematic difference in error patterns between machines and humans suggests that humans use distinct, non-interpretable features not captured by standard machine representations.