[Paper Review] Low-Latency Video Semantic Segmentation

Yule Li, Jianping Shi+1 more|ArXiv.org|Apr 2, 2018

Visual Attention and Saliency Detection25 references3 citations

TL;DR

This paper proposes a low-latency video semantic segmentation framework using adaptive feature propagation via spatially variant convolution and an adaptive scheduler for keyframe selection. By dynamically allocating computation based on accuracy prediction and efficiently reusing features across frames, the method reduces latency to 119ms on Cityscapes—33% of the baseline—while achieving state-of-the-art performance with improved accuracy and efficiency.

ABSTRACT

Recent years have seen remarkable progress in semantic segmentation. Yet, it remains a challenging task to apply segmentation techniques to video-based applications. Specifically, the high throughput of video streams, the sheer cost of running fully convolutional networks, together with the low-latency requirements in many real-world applications, e.g. autonomous driving, present a significant challenge to the design of the video segmentation framework. To tackle this combined challenge, we develop a framework for video semantic segmentation, which incorporates two novel components: (1) a feature propagation module that adaptively fuses features over time via spatially variant convolution, thus reducing the cost of per-frame computation; and (2) an adaptive scheduler that dynamically allocate computation based on accuracy prediction. Both components work together to ensure low latency while maintaining high segmentation quality. On both Cityscapes and CamVid, the proposed framework obtained competitive performance compared to the state of the art, while substantially reducing the latency, from 360 ms to 119 ms.

Motivation & Objective

• Address the challenge of low-latency video semantic segmentation in real-time applications like autonomous driving.
• Overcome the limitations of existing methods that reduce amortized cost but not maximum latency.
• Improve feature reuse across frames by adapting propagation weights spatially, rather than uniformly.
• Design a dynamic keyframe scheduling mechanism that reduces computation without sacrificing accuracy.
• Achieve a better trade-off between inference latency and segmentation accuracy than prior approaches.

Proposed method

• Introduces a spatially variant convolution-based feature propagation module that adaptively fuses features from previous frames using location-specific weights.
• Employs an adaptive scheduler that predicts segmentation accuracy and triggers keyframe computation only when necessary.
• Uses a two-branch network architecture: a lower part for fast inference and a higher part for accurate feature extraction.
• Applies parallel processing to coordinate keyframe computation and feature propagation, minimizing idle time.
• Integrates the adaptive propagation and scheduling modules into an end-to-end trainable framework for online video segmentation.
• Leverages intermediate feature maps for accuracy prediction to guide dynamic scheduling decisions.

Experimental results

Research questions

• RQ1 Can adaptive feature propagation using spatially variant convolution improve segmentation accuracy while reducing computation in video streams?
• RQ2 Can a dynamic keyframe scheduling strategy reduce maximum latency without degrading performance?
• RQ3 How does the proposed framework compare to state-of-the-art methods in terms of latency and mIoU on benchmark datasets?
• RQ4 To what extent does the adaptive scheduler reduce the number of keyframes while maintaining high accuracy?
• RQ5 Does the combination of adaptive propagation and scheduling enable real-time performance in complex, dynamic environments?

Key findings

• The proposed framework reduces inference latency to 119ms on the Cityscapes dataset, representing a 67% reduction from the baseline of 360ms.
• On Cityscapes, the method achieves a mean Intersection over Union (mIoU) of 82.9%, outperforming previous state-of-the-art methods.
• On the CamVid dataset, the method achieves a pixel accuracy of 94.6% and a class accuracy of 82.9%, both the highest among compared methods.
• The adaptive scheduler consistently outperforms fixed-rate and threshold-based scheduling across all key-frame intervals, improving accuracy under constrained computation.
• Latency analysis shows that the adaptive propagation and scheduling modules contribute only 10.5% and 5.5% of the total latency, respectively, indicating high efficiency.
• Visual results on both Cityscapes and CamVid demonstrate significant improvements in boundary and texture detail, especially in dynamic or complex scenes.