[Paper Review] Inductive Guided Filter: Real-time Deep Image Matting with Weakly Annotated Masks on Mobile Devices

Yaoyi Li, Jianfu Zhang+2 more|arXiv (Cornell University)|May 16, 2019

Image Enhancement Techniques36 references4 citations

TL;DR

This paper proposes Inductive Guided Filter, a real-time deep image matting method for mobile devices that uses a lightweight hourglass network to parameterize the Guided Filter with weakly annotated masks as input. It introduces a Gabor loss for enhanced texture modeling and achieves high accuracy and speed, running at over 5000 FPS on GPU and under 26ms on iPhone SE, outperforming prior methods in efficiency and robustness to noisy masks.

ABSTRACT

Recently, significant progress has been achieved in deep image matting. Most of the classical image matting methods are time-consuming and require an ideal trimap which is difficult to attain in practice. A high efficient image matting method based on a weakly annotated mask is in demand for mobile applications. In this paper, we propose a novel method based on Deep Learning and Guided Filter, called Inductive Guided Filter, which can tackle the real-time general image matting task on mobile devices. We design a lightweight hourglass network to parameterize the original Guided Filter method that takes an image and a weakly annotated mask as input. Further, the use of Gabor loss is proposed for training networks for complicated textures in image matting. Moreover, we create an image matting dataset MAT-2793 with a variety of foreground objects. Experimental results demonstrate that our proposed method massively reduces running time with robust accuracy.

Motivation & Objective

• To develop a real-time, efficient deep image matting framework suitable for mobile deployment.
• To reduce dependency on high-quality trimaps by enabling robust performance with weakly annotated masks.
• To improve texture detail preservation in matting through a novel loss function.
• To create a large-scale, diverse image matting dataset (MAT-2793) for training and evaluation.
• To achieve high inference speed without sacrificing accuracy on mobile and edge devices.

Proposed method

• A lightweight hourglass network is designed to parameterize the original Guided Filter, enabling end-to-end learning with image and mask inputs.
• The model uses a dual-branch architecture sharing a single lightweight backbone to predict linear coefficients A and B for alpha matte estimation.
• A novel Gabor loss is introduced, applying a bundle of Gabor filters to extract high-frequency texture features and improve detail fidelity.
• The network is trained using a GAN-based framework with adversarial loss, L1 loss, and the proposed Gabor loss for enhanced realism and detail.
• The fast version of the Guided Filter with a downsampling factor of 4 is used for inference acceleration, mimicking the original filter's efficiency.
• Weakly annotated masks are generated via random dilation and erosion on foreground masks, simulating real-world user input noise.

Experimental results

Research questions

• RQ1 Can a deep learning-based method achieve real-time inference for image matting on mobile devices?
• RQ2 How can a model be made robust to weakly annotated masks without requiring precise trimaps?
• RQ3 Can a novel loss function based on Gabor filters improve texture detail preservation in image matting?
• RQ4 What level of performance and speed can be achieved with a lightweight architecture on mobile hardware?
• RQ5 How does the proposed method compare to state-of-the-art models in terms of accuracy and inference latency?

Key findings

• The proposed method achieves inference speeds of over 5000 FPS on a single NVIDIA V100 GPU with a batch size of 256, demonstrating real-time capability.
• On an iPhone SE (2016), the model runs in 25.9ms per image, achieving real-time performance in production.
• The model achieves robust performance on both MAT-2793 and Adobe Composition-1k datasets, even with noisy or inexact input masks.
• The Gabor loss significantly improves texture detail preservation, as shown in qualitative comparisons with baseline methods.
• The method outperforms Adobe DIM and other SOTA models in speed while maintaining competitive accuracy, especially in the presence of weakly annotated masks.
• The MAT-2793 dataset, containing 2793 diverse foreground objects, is the largest publicly available image matting dataset to date.