[Paper Review] Towards Real-Time Automatic Portrait Matting on Mobile Devices

Seokjun Seo, Seungwoo Choi+5 more|arXiv (Cornell University)|Apr 8, 2019

Advanced Image and Video Retrieval Techniques34 references4 citations

TL;DR

This paper proposes MMNet, a lightweight, real-time portrait matting model for mobile devices using multi-branch dilated convolutions and linear bottleneck blocks. It achieves 30 FPS on Xiaomi Mi 5 with only 15% higher gradient error than state-of-the-art models, while using an order of magnitude fewer parameters than Mobile DeepLabv3.

ABSTRACT

We tackle the problem of automatic portrait matting on mobile devices. The proposed model is aimed at attaining real-time inference on mobile devices with minimal degradation of model performance. Our model MMNet, based on multi-branch dilated convolution with linear bottleneck blocks, outperforms the state-of-the-art model and is orders of magnitude faster. The model can be accelerated four times to attain 30 FPS on Xiaomi Mi 5 device with moderate increase in the gradient error. Under the same conditions, our model has an order of magnitude less number of parameters and is faster than Mobile DeepLabv3 while maintaining comparable performance. The accompanied implementation can be found at \url{https://github.com/hyperconnect/MMNet}.

Motivation & Objective

• Address the challenge of achieving real-time inference for automatic portrait matting on mobile devices.
• Reduce model size and computational cost without sacrificing matting quality.
• Overcome the limitations of existing methods that rely on user-interactive inputs like trimaps or scribbles.
• Develop a compact neural network that maintains high performance under strict latency and parameter constraints.
• Enable deployment of real-time portrait matting in practical mobile applications such as photo editing and video processing.

Proposed method

• Adopt an encoder-decoder architecture with skip connections to preserve spatial details.
• Use depthwise separable convolutions as the basic operation to reduce FLOPs and model size.
• Introduce multi-branch dilated convolutions with different dilation rates to capture multi-scale context efficiently.
• Employ linear bottleneck blocks to enhance representational capacity while maintaining efficiency.
• Apply a width multiplier to control model capacity, latency, and parameter count for mobile deployment.
• Incorporate a novel gradient loss and multiple auxiliary losses to improve alpha matte quality.

Experimental results

Research questions

• RQ1 Can a compact neural network achieve real-time portrait matting on mobile devices without significant performance degradation?
• RQ2 How effective are multi-branch dilated convolutions with linear bottleneck blocks in balancing accuracy and efficiency?
• RQ3 What is the trade-off between model size, latency, and matting quality on mobile hardware?
• RQ4 To what extent can quantization and model compression improve inference speed with minimal accuracy loss?
• RQ5 Can the proposed architecture generalize beyond portrait matting to other image matting or video matting tasks?

Key findings

• MMNet achieves 30 FPS on Xiaomi Mi 5 with only 15% higher gradient error than the state-of-the-art, enabling real-time performance.
• The model uses an order of magnitude fewer parameters than Mobile DeepLabv3 while maintaining comparable performance.
• Quantization to 8-bit reduces latency by 25% and slightly improves gradient error, demonstrating robustness to low-precision inference.
• Ablation studies confirm that multi-branch dilated convolutions and linear bottleneck blocks are critical for performance and efficiency.
• The model outperforms all prior methods in the Pareto front of latency, parameter count, and gradient error on mobile devices.
• The implementation is publicly available at https://github.com/hyperconnect/MMNet for reproducibility and further development.