[Paper Review] Boundary-Aware Network for Fast and High-Accuracy Portrait Segmentation

Xi Chen, Donglian Qi+1 more|arXiv (Cornell University)|Jan 12, 2019

Advanced Neural Network Applications25 references18 citations

TL;DR

This paper proposes Boundary-Aware Network (BANet), a lightweight, real-time portrait segmentation model that enhances boundary detail extraction using a boundary attention mechanism and a novel refine loss function. BANet achieves 43 FPS on 512×512 images and produces segmentation results finer than human-annotated masks, outperforming prior methods in detail quality despite minimal parameters (0.62 MB).

ABSTRACT

Compared with other semantic segmentation tasks, portrait segmentation requires both higher precision and faster inference speed. However, this problem has not been well studied in previous works. In this paper, we propose a lightweight network architecture, called Boundary-Aware Network (BANet) which selectively extracts detail information in boundary area to make high-quality segmentation output with real-time( >25FPS) speed. In addition, we design a new loss function called refine loss which supervises the network with image level gradient information. Our model is able to produce finer segmentation results which has richer details than annotations.

Motivation & Objective

• Address the lack of real-time, high-accuracy portrait segmentation models that preserve fine details like hair and clothing edges.
• Overcome the limitations of coarse human annotations and traditional semantic segmentation models that downsample feature maps, losing detail.
• Design a lightweight network architecture suitable for mobile deployment while maintaining state-of-the-art boundary fidelity.
• Introduce a novel training strategy that improves detail quality beyond annotation limits using image gradient supervision.

Proposed method

• Propose a two-branch network architecture: a high-level semantic branch and a low-level feature mining branch guided by a boundary attention map.
• Design a boundary attention mechanism that selectively focuses on edge regions by leveraging high-level features to attend to low-level details.
• Introduce a refine loss function that uses image-level gradient information to supervise boundary refinement, enabling the model to learn finer details than the annotations.
• Optimize the network for efficiency with depthwise separable convolutions and channel reduction, achieving only 0.62 MB parameters.
• Use a multi-scale training and inference strategy to maintain performance across different input resolutions.
• Train the model end-to-end on the Supervise.ly dataset, leveraging its high-quality annotations as a strong baseline.

Experimental results

Research questions

• RQ1 Can a lightweight segmentation network achieve real-time inference speed (≥25 FPS) while producing higher-quality boundaries than human-annotated masks?
• RQ2 How effective is a boundary attention mechanism in selectively enhancing feature extraction at object boundaries without increasing model complexity?
• RQ3 To what extent can a refine loss function based on image gradients improve detail quality beyond the resolution of ground-truth annotations?
• RQ4 Can a model trained on limited data (1,400 images) generalize well enough for mobile deployment while outperforming larger models in boundary fidelity?
• RQ5 How does the combination of boundary attention and refine loss compare to standard loss functions in terms of mIoU and visual detail quality?

Key findings

• BANet-64 achieves 43 FPS on 512×512 images with only 0.62 MB parameters, significantly outperforming PFCN+ (134.27 MB) and U-Net (13.40 MB) in model efficiency.
• Despite a slightly lower mIoU (95.80%) than PFCN+ (95.92%), BANet-64 produces segmentation results with finer boundaries than KNN-annotated masks, indicating superior detail quality.
• The ablation study confirms that boundary attention improves mIoU by 0.3% and refine loss by 0.6%, with visual results showing marked improvement in edge sharpness.
• BANet-512 achieves 96.13% mIoU with 12.75 MB parameters, outperforming other models in accuracy while maintaining real-time inference at 29 FPS.
• The model's ability to generate results finer than annotations demonstrates that the refine loss effectively learns details beyond the annotation quality.
• On mobile deployment, BANet-64's small size and high speed make it feasible for real-time applications like background replacement and image beautification.