[Paper Review] Fast Video Salient Object Detection via Spatiotemporal Knowledge Distillation

Yi Tang, Yuan Li+1 more|arXiv (Cornell University)|Oct 20, 2020

Visual Attention and Saliency Detection40 references4 citations

TL;DR

This paper proposes a lightweight video salient object detection network using spatiotemporal knowledge distillation to achieve high accuracy without complex sequential modules. By combining spatial distillation with saliency-guided feature embedding and temporal distillation via infer-frame feature encoding, the method achieves state-of-the-art performance with only 0.01 seconds per frame inference time.

ABSTRACT

Since the wide employment of deep learning frameworks in video salient object detection, the accuracy of the recent approaches has made stunning progress. These approaches mainly adopt the sequential modules, based on optical flow or recurrent neural network (RNN), to learn robust spatiotemporal features. These modules are effective but significantly increase the computational burden of the corresponding deep models. In this paper, to simplify the network and maintain the accuracy, we present a lightweight network tailored for video salient object detection through the spatiotemporal knowledge distillation. Specifically, in the spatial aspect, we combine a saliency guidance feature embedding structure and spatial knowledge distillation to refine the spatial features. In the temporal aspect, we propose a temporal knowledge distillation strategy, which allows the network to learn the robust temporal features through the infer-frame feature encoding and distilling information from adjacent frames. The experiments on widely used video datasets (e.g., DAVIS, DAVSOD, SegTrack-V2) prove that our approach achieves competitive performance. Furthermore, without the employment of the complex sequential modules, the proposed network can obtain high efficiency with 0.01s per frame.

Motivation & Objective

• To address the high computational cost of existing video salient object detection models that rely on complex sequential modules like optical flow or RNNs.
• To maintain high detection accuracy while significantly reducing model complexity and inference time.
• To develop a lightweight network that leverages knowledge distillation to extract robust spatiotemporal features without external sequential modules.
• To validate the effectiveness of spatial and temporal knowledge distillation components in improving feature representation for video saliency.

Proposed method

• Introduces a saliency guidance feature embedding module to enhance spatial feature representation by focusing on salient regions.
• Employs spatial knowledge distillation to refine features by transferring knowledge from a teacher-like feature map to the student network.
• Proposes temporal knowledge distillation that uses adjacent frame features as supervision to guide the current frame's feature learning.
• Incorporates an infer-frame feature encoding module using a mutual attention block to fuse high-level features from neighboring frames.
• Uses a weighted combination of cross-entropy loss and distillation loss with a balance factor α = 0.7 to optimize training.
• Applies self-distillation by leveraging the network’s own predictions as supervision, reducing the need for external teacher models.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively applied to video salient object detection to reduce model complexity without sacrificing accuracy?
• RQ2 How can spatiotemporal features be effectively learned in a lightweight network without using optical flow or RNNs?
• RQ3 What is the optimal way to encode and transfer information from adjacent frames to improve temporal modeling?
• RQ4 How does the balance between cross-entropy and distillation loss affect the final performance?

Key findings

• The proposed method achieves an F-measure of 0.883 and MAE of 0.022 on the DAVIS dataset, outperforming baseline models without sequential modules.
• The addition of spatial knowledge distillation improves F-measure by 1.9% and reduces MAE by 1.4% compared to the baseline.
• The temporal distillation with infer-frame feature encoding, especially using the 'Multiply' operation or mutual attention, boosts performance significantly, achieving F-measure 0.882–0.883.
• The optimal balance factor α for loss weighting is 0.7, as it yields the best trade-off between cross-entropy and distillation loss.
• The inference time is reduced to only 0.01 seconds per frame, demonstrating high efficiency despite competitive accuracy.
• The infer-frame feature encoding module can be removed at test time without performance degradation, enhancing inference speed.