[Paper Review] Deep Learning Face Attributes in the Wild

Ziwei Liu, Ping Luo+2 more|arXiv (Cornell University)|Nov 28, 2014

Face recognition and analysisComputer Science25 references103 citations

TL;DR

This paper proposes a two-stream deep learning framework, LNets+ANet, that jointly fine-tunes two CNNs—LNet for weakly supervised face localization using image-level attribute tags and ANet for attribute prediction—after pre-training on general objects and face identities, respectively. The method achieves state-of-the-art performance on CelebA and LFWA, outperforming prior work by up to 10% accuracy, and reveals that pre-training enables discovery of semantic concepts and enables localization without bounding boxes.

ABSTRACT

Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts.

Motivation & Objective

• To address the challenge of face attribute prediction in unconstrained, real-world images with complex variations such as pose, lighting, and occlusion.
• To develop a framework that enables face localization and attribute prediction without requiring bounding boxes or landmarks during training.
• To investigate how pre-training on massive object categories and face identities improves feature learning for face localization and attribute recognition.
• To enable efficient, real-time inference on arbitrary-sized images using a novel fast feed-forward scheme for locally shared filters.
• To uncover implicit semantic concepts learned by deep networks during pre-training and fine-tuning.

Proposed method

• LNet is pre-trained on ImageNet for general object recognition to learn robust features for face localization, then fine-tuned using only image-level attribute tags without bounding boxes.
• ANet is pre-trained on a large-scale face identification dataset to learn identity-related features, then fine-tuned with attribute tags for attribute prediction.
• A novel interweaved feed-forward operation enables efficient inference on arbitrary-sized images by replacing patch-by-patch evaluation, reducing redundant computation for locally shared filters.
• Face localization is achieved by averaging response maps of LNet’s filters, which show strong spatial activation at face locations even without explicit localization supervision.
• Semantic concept discovery is analyzed via clustering of the fully connected layer weights in ANet, revealing interpretable groupings of attributes such as co-occurrence patterns and appearance traits.
• The framework uses a cascaded architecture where LNet provides rough face region localization, and ANet extracts features from the localized region for final attribute prediction.

Experimental results

Research questions

• RQ1 Can face localization be effectively learned using only image-level attribute tags, without requiring bounding boxes or landmarks?
• RQ2 How does pre-training on massive general object categories improve face localization performance in weakly supervised settings?
• RQ3 To what extent do high-level neurons in a face-identity-pretrained network (ANet) automatically discover semantic concepts like gender, age, or race?
• RQ4 Can a fast, single-pass inference method be designed for locally shared filters to enable real-time processing of arbitrary-sized inputs?
• RQ5 How well does the proposed framework generalize to unseen attributes when fine-tuned on a small subset of new labels?

Key findings

• The proposed LNets+ANet framework achieves 87% accuracy on CelebA and 84% on LFWA, outperforming PANDA-l (81%) and FaceTracer (81%) by 3–10 percentage points, even when PANDA-l uses ground-truth bounding boxes.
• LNet achieves strong face localization performance using only image-level tags—its response maps localize faces effectively, demonstrating that weak supervision can yield robust localization features.
• Pre-training ANet on massive face identities enables it to discover meaningful semantic concepts such as race, gender, and age, which are later refined during attribute fine-tuning.
• The weight matrix of ANet’s fully connected layer reveals interpretable attribute groupings: e.g., Group #1 shows strong co-occurrence of 'Attractive' and 'Heavy Makeup', while Group #3 reflects color-based similarities.
• When tested on 30 new attributes in LFWA+, LNets+ANet achieves 8%, 10%, and 3% average accuracy gains over FaceTracer, PANDA-w, and PANDA-l, respectively, demonstrating strong generalization.
• The method reduces inference time to 35ms for localization and 14ms for feature extraction on a 300×300 image, outperforming patch-by-patch scanning (80ms), making it suitable for real-time applications.