[Paper Review] EasyPortrait -- Face Parsing and Portrait Segmentation Dataset

Karina Kvanchiani, Petrova, Elizaveta+3 more|arXiv (Cornell University)|Apr 26, 2023

Face recognition and analysis4 citations

TL;DR

This paper introduces EasyPortrait, a large-scale, high-quality dataset of 20,000 indoor portrait images with fine-grained 9-class semantic segmentation masks for face parsing and portrait segmentation. The dataset is created via crowdsourcing with expert-verified labeling, excluding teeth and beard from skin, enabling precise applications like skin enhancement and teeth whitening, and achieving state-of-the-art mIoU scores on multiple models with minimal training effort.

ABSTRACT

Recently, video conferencing apps have become functional by accomplishing such computer vision-based features as real-time background removal and face beautification. Limited variability in existing portrait segmentation and face parsing datasets, including head poses, ethnicity, scenes, and occlusions specific to video conferencing, motivated us to create a new dataset, EasyPortrait, for these tasks simultaneously. It contains 40,000 primarily indoor photos repeating video meeting scenarios with 13,705 unique users and fine-grained segmentation masks separated into 9 classes. Inappropriate annotation masks from other datasets caused a revision of annotator guidelines, resulting in EasyPortrait's ability to process cases, such as teeth whitening and skin smoothing. The pipeline for data mining and high-quality mask annotation via crowdsourcing is also proposed in this paper. In the ablation study experiments, we proved the importance of data quantity and diversity in head poses in our dataset for the effective learning of the model. The cross-dataset evaluation experiments confirmed the best domain generalization ability among portrait segmentation datasets. Moreover, we demonstrate the simplicity of training segmentation models on EasyPortrait without extra training tricks. The proposed dataset and trained models are publicly available.

Motivation & Objective

• Address the lack of diverse, high-quality, real-world portrait datasets for training lightweight, real-time models in video conferencing applications.
• Overcome limitations in existing datasets where beard, teeth, and glasses are incorrectly included in skin or mouth masks.
• Provide a clean, manually annotated dataset using crowdsourcing with expert validation to ensure label quality and transparency.
• Enable precise facial enhancement tasks such as teeth whitening, skin smoothing, and red-eye removal by separating key facial components into distinct classes.
• Facilitate rapid model training with high performance using standard architectures without complex data augmentation or self-supervised pretraining.

Proposed method

• Collected 20,000 images from 8,377 unique users via two crowdsourcing platforms, primarily in indoor settings like homes, offices, or outdoors with smartphones or laptops.
• Generated 9-class semantic segmentation masks: background, person, face skin, left/right eyebrow, left/right eye, lips, and teeth, with strict labeling rules excluding beard and nostrils from skin.
• Employed a two-stage annotation pipeline: initial labeling by crowdworkers followed by expert verification to ensure high-quality, consistent segmentation masks.
• Used standard data augmentation techniques including image padding, resizing to 1024×1024, photometric distortion (brightness, contrast, saturation, hue), and normalization.
• Trained multiple state-of-the-art segmentation models (e.g., SegFormer, FPN, BiSeNet-V2) on the dataset using the MMSegmentation framework with fixed hyperparameters and AdamW optimizer.
• Evaluated models using mean Intersection over Union (mIoU) across all classes, with no data-specific fine-tuning or training tricks.

Figure 1: The face parsing and portrait segmentation annotation examples from the EasyPortrait dataset.

Experimental results

Research questions

• RQ1 Can a large-scale, real-world portrait dataset with fine-grained, expert-verified segmentation masks improve performance in face parsing and portrait segmentation tasks without synthetic data or self-supervised pretraining?
• RQ2 How does excluding teeth and beard from the skin class affect the performance of downstream tasks like teeth whitening and skin enhancement?
• RQ3 To what extent can standard deep learning models achieve high mIoU scores on a real-world portrait dataset with only 20,000 images and no complex data augmentation?
• RQ4 Can a crowdsourced pipeline with expert validation produce a dataset with sufficient quality and diversity to support robust model generalization in real-time video applications?
• RQ5 How do different backbone architectures (e.g., ResNet-50, MobileNet-V2, SegFormer-B5) perform on a dataset with diverse occlusions and facial features?

Key findings

• The FPN + ResNet-50 model fine-tuned on 1024×1024 crops achieved the highest mIoU score of 84.08 on the EasyPortrait dataset, demonstrating strong performance with minimal architectural complexity.
• SegFormer-B5 with 1024×1024 input achieved an mIoU of 84.94, showing that efficient transformers can achieve high accuracy on portrait segmentation with limited data.
• Even lightweight models like FCN + MobileNet-V2 (1024) achieved mIoU scores above 70 on key classes such as eyes (70.81) and lips (69.99), indicating strong generalization on small models.
• The BiSeNet-V2 model achieved 78.39 mIoU on the full dataset, showing that lightweight architectures can perform well when trained on high-quality, diverse data.
• The dataset’s design—separating teeth and excluding beard from skin—enabled precise modeling of facial enhancement tasks, such as teeth whitening and skin smoothing, which are not feasible with standard datasets.
• All models trained on EasyPortrait achieved high mIoU values (e.g., >95 for face skin and person classes) in just 20,000 iterations, proving the dataset’s suitability for rapid, high-accuracy training.

Figure 2: Visual comparison of existing face parsing datasets. Only Face Synthetics and EasyPortrait datasets can be used to solve background removal and face enhancement problems. None of them except the last one can be used in the teeth whitening task. We are not including LFW-PL and FaceOcc datas