[Paper Review] Structure-Aware Network for Lane Marker Extraction with Dynamic Vision Sensor

Wensheng Cheng, Hao Luo+3 more|arXiv (Cornell University)|Aug 14, 2020

Autonomous Vehicle Technology and Safety45 references17 citations

TL;DR

This paper proposes a structure-aware network (SANet) with multidirectional slice convolution for lane marker extraction from high-resolution dynamic vision sensor (DVS) images. It introduces the DET dataset—5,424 DVS images at 1280×800 resolution with pixel-level annotations—and demonstrates that SANet outperforms state-of-the-art methods, achieving 74.21% mean F1 and 60.86% mean IoU on the DET benchmark.

ABSTRACT

Lane marker extraction is a basic yet necessary task for autonomous driving. Although past years have witnessed major advances in lane marker extraction with deep learning models, they all aim at ordinary RGB images generated by frame-based cameras, which limits their performance in extreme cases, like huge illumination change. To tackle this problem, we introduce Dynamic Vision Sensor (DVS), a type of event-based sensor to lane marker extraction task and build a high-resolution DVS dataset for lane marker extraction. We collect the raw event data and generate 5,424 DVS images with a resolution of 1280$ imes$800 pixels, the highest one among all DVS datasets available now. All images are annotated with multi-class semantic segmentation format. We then propose a structure-aware network for lane marker extraction in DVS images. It can capture directional information comprehensively with multidirectional slice convolution. We evaluate our proposed network with other state-of-the-art lane marker extraction models on this dataset. Experimental results demonstrate that our method outperforms other competitors. The dataset is made publicly available, including the raw event data, accumulated images and labels.

Motivation & Objective

• Address the limitations of RGB-based lane detection in extreme lighting conditions such as rapid illumination changes.
• Overcome the poor performance of frame-based cameras in low-light or high-dynamic-range scenarios by leveraging Dynamic Vision Sensors (DVS).
• Develop a high-resolution DVS dataset (DET) with 5,424 images at 1280×800 resolution and multi-class semantic segmentation labels for lane marker extraction.
• Design a structure-aware network (SANet) that effectively captures directional and contextual structural information in DVS data, which lacks rich appearance cues.
• Improve lane marker segmentation accuracy by incorporating multidirectional slice convolution to model long, continuous lane structures.

Proposed method

• Constructed the DET dataset using a CeleX V DVS camera, collecting 5 hours of raw event streams and generating 5,424 high-resolution DVS images at 1280×800 pixels.
• Provided two annotation formats: multi-class semantic segmentation and binary segmentation, to support both general and specialized lane detection models.
• Proposed a structure-aware network (SANet) that uses a Multidirectional Slice Convolution (MSC) module to capture directional features across multiple orientations.
• Integrated slice convolution with varying kernel sizes (3 to 11) to model structural continuity of lane markers, with optimal performance at kernel size 9.
• Employed a multi-scale skip connection design to preserve spatial details while aggregating contextual features across layers.
• Trained and evaluated the model using standard metrics: mean F1 score and mean Intersection over Union (IoU), comparing against FCN, DeepLabv3, RefineNet, and SCNN.

Experimental results

Research questions

• RQ1 Can a high-resolution DVS dataset significantly improve the performance of lane marker extraction in extreme lighting conditions?
• RQ2 How effective is a structure-aware network with multidirectional slice convolution in capturing long, continuous lane structures in DVS data?
• RQ3 Does incorporating directional feature learning via slice convolution outperform standard CNNs and specialized models like SCNN on event-based lane segmentation?
• RQ4 What is the optimal kernel size for slice convolution in modeling lane marker geometry in DVS images?
• RQ5 To what extent does the proposed method generalize compared to existing semantic segmentation and lane-specific models on event-based data?

Key findings

• The proposed DET dataset is the first high-resolution DVS dataset for lane marker extraction, with 5,424 images at 1280×800 resolution and detailed pixel-level annotations.
• The structure-aware network with multidirectional slice convolution (SANet_MSC) achieved 74.21% mean F1 and 60.86% mean IoU, outperforming SCNN (73.85% F1, 60.44% IoU) by 0.42% on IoU.
• The MSC module significantly improved performance over baseline models, with a 1.78% increase in mean F1 and 1.91% increase in mean IoU compared to the baseline.
• Kernel size 9 in slice convolution yielded the best performance (74.21% F1, 60.86% IoU), outperforming both smaller (3–7) and larger (11) kernels.
• The model demonstrated superior robustness in extreme lighting conditions, such as tunnel transitions, where frame-based cameras fail due to low dynamic range.
• The proposed method outperformed general semantic segmentation models (FCN, DeepLabv3, RefineNet), which achieved lower than 65% F1, due to lack of structural inductive bias for lane geometry.