[Paper Review] Highway Driving Dataset for Semantic Video Segmentation

Byungju Kim, Junho Yim+1 more|arXiv (Cornell University)|Nov 2, 2020

Advanced Image and Video Retrieval Techniques15 citations

TL;DR

This paper introduces the Highway Driving dataset, a temporally and spatially dense video benchmark for semantic video segmentation, with 20 high-frame-rate (30Hz) sequences fully annotated at the pixel level. It proposes a lightweight, recurrent baseline model that leverages temporal correlation to achieve real-time inference with competitive accuracy, significantly reducing runtime while maintaining performance through priming frequency control.

ABSTRACT

Scene understanding is an essential technique in semantic segmentation. Although there exist several datasets that can be used for semantic segmentation, they are mainly focused on semantic image segmentation with large deep neural networks. Therefore, these networks are not useful for real time applications, especially in autonomous driving systems. In order to solve this problem, we make two contributions to semantic segmentation task. The first contribution is that we introduce the semantic video dataset, the Highway Driving dataset, which is a densely annotated benchmark for a semantic video segmentation task. The Highway Driving dataset consists of 20 video sequences having a 30Hz frame rate, and every frame is densely annotated. Secondly, we propose a baseline algorithm that utilizes a temporal correlation. Together with our attempt to analyze the temporal correlation, we expect the Highway Driving dataset to encourage research on semantic video segmentation.

Motivation & Objective

• Address the lack of temporally dense video segmentation datasets for autonomous driving scenarios.
• Enable real-time semantic video segmentation by reducing inference time without sacrificing accuracy.
• Provide a benchmark dataset with high temporal and spatial annotation density for video segmentation research.
• Develop a baseline algorithm that exploits temporal correlation to accelerate inference.
• Demonstrate that temporal consistency can be leveraged to reduce reliance on expensive, deep networks per frame.

Proposed method

• Collect 20 high-frame-rate (30Hz) video sequences from highway driving scenarios with dense pixel-level annotations.
• Annotate frames sequentially to ensure temporal consistency across adjacent frames.
• Propose a two-branch network: a slow, accurate priming network and a fast, lightweight approximating network.
• Use a recurrent framework to propagate predictions from the priming network to subsequent frames via the approximating network.
• Control inference speed by adjusting the frequency of priming network inference, balancing accuracy and runtime.
• Train the approximating network to mimic the priming network's output using knowledge distillation and temporal consistency loss.

Experimental results

Research questions

• RQ1 Can a video segmentation model achieve high accuracy with minimal runtime by exploiting temporal correlation?
• RQ2 How does temporal density in annotations affect the performance and reliability of semantic video segmentation models?
• RQ3 To what extent can a lightweight network maintain accuracy when guided by periodic predictions from a deeper, more accurate network?
• RQ4 Is it possible to design a real-time semantic video segmentation system that maintains performance comparable to state-of-the-art models?
• RQ5 How does the priming frequency affect the trade-off between inference speed and segmentation accuracy?

Key findings

• The Highway Driving dataset provides 20 video sequences with 30Hz frame rates and fully dense pixel-level annotations, offering higher temporal density than existing datasets like CamVid (1Hz).
• The proposed baseline algorithm achieves comparable performance to stronger models while significantly reducing runtime, with performance controllable via priming frequency.
• The algorithm maintains high-quality predictions even after 59 frames without re-priming, demonstrating effective temporal propagation through the recurrent framework.
• Qualitative results show that the model preserves scene consistency and accurately segments moving objects, especially when dense temporal annotations are available.
• The algorithm generalizes well, as verified on the CamVid dataset, showing consistent performance across different datasets without fine-tuning.
• Runtime is normalized relative to DRN; the baseline achieves faster inference than standard models while maintaining competitive accuracy, proving the effectiveness of temporal correlation in reducing computational load.