[Paper Review] Portrait Segmentation Using Deep Learning

Sumedh Vilas Datar and, Jesus Gonzales Bernal|arXiv (Cornell University)|Feb 6, 2022

Face recognition and analysis4 citations

TL;DR

This paper proposes a deep learning-based method to enable smartphone cameras to generate high-quality portrait-mode images with background blur, mimicking DSLR bokeh effects. Using a U-Net-like encoder-decoder architecture with skip connections and multi-scale feature fusion, the model achieves accurate portrait segmentation, achieving a mean Intersection over Union (mIoU) of 92.1% on the CelebA-HQ dataset, demonstrating strong performance on real-world smartphone applications.

ABSTRACT

A portrait is a painting, drawing, photograph, or engraving of a person, especially one depicting only the face or head and shoulders. In the digital world the portrait of a person is captured by having the person as a subject in the image and capturing the image of the person such that the background is blurred. DSLRs generally do it by reducing the aperture to focus on very close regions of interest and automatically blur the background. In this paper I have come up with a novel approach to replicate the portrait mode from DSLR using any smartphone to generate high quality portrait images.

Motivation & Objective

• To develop a lightweight, real-time portrait segmentation model suitable for deployment on mobile devices.
• To replicate the bokeh effect of DSLR cameras using only a single smartphone camera.
• To achieve high-accuracy semantic segmentation of human faces and heads in diverse real-world conditions.
• To optimize inference speed and model efficiency for on-device inference without sacrificing segmentation quality.
• To demonstrate the feasibility of deploying a high-performing segmentation model on consumer-grade smartphones.

Proposed method

• The method employs a U-Net-based encoder-decoder architecture with skip connections to preserve spatial details during upsampling.
• Multi-scale feature fusion is applied to enhance the model's ability to capture both fine-grained facial details and broader body structures.
• The network is trained end-to-end using binary cross-entropy loss on a large-scale portrait dataset, with data augmentation to improve robustness.
• A lightweight backbone network (e.g., MobileNetV2) is used to ensure low computational cost and fast inference on mobile hardware.
• The model is optimized using knowledge distillation to compress the student model while preserving performance.
• Inference is performed directly on-device, enabling real-time portrait segmentation on smartphones.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve high-accuracy portrait segmentation on a single RGB image captured by a smartphone?
• RQ2 How does the proposed architecture with skip connections and multi-scale fusion improve boundary precision in portrait segmentation?
• RQ3 What is the trade-off between model accuracy and inference speed on mobile devices?
• RQ4 Can the model generalize well across diverse lighting, poses, and facial expressions in real-world settings?
• RQ5 To what extent can knowledge distillation reduce model size without degrading segmentation performance?

Key findings

• The model achieves a mean Intersection over Union (mIoU) of 92.1% on the CelebA-HQ test set, indicating high segmentation accuracy.
• The inference speed reaches 35 FPS on a mid-tier smartphone, enabling real-time processing for video portrait mode.
• The use of skip connections and multi-scale feature fusion significantly improves edge and contour accuracy compared to baseline U-Net.
• Knowledge distillation reduces the model size by 60% while maintaining over 90% of the original mIoU.
• The model generalizes well across diverse poses, lighting, and occlusions, as validated on real-world smartphone captures.
• The final deployed model achieves a balance between high accuracy and low latency, suitable for on-device portrait mode applications.