[Paper Review] Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network

Batool Salehi, Guillem Reus-Muns+6 more|arXiv (Cornell University)|Jan 12, 2022

Millimeter-Wave Propagation and Modeling4 citations

TL;DR

This paper proposes a deep learning-based beam selection framework for mmWave vehicular networks that leverages multimodal sensor data (LiDAR, camera, GPS) at the wireless edge to accelerate beam sweeping. By using fused, edge-deployed models to predict top-K beam pairs, the system achieves 95–96% faster beam selection than RF-only methods and outperforms state-of-the-art techniques by 20–22% in top-10 beam pair accuracy.

ABSTRACT

Beam selection for millimeter-wave links in a vehicular scenario is a challenging problem, as an exhaustive search among all candidate beam pairs cannot be assuredly completed within short contact times. We solve this problem via a novel expediting beam selection by leveraging multimodal data collected from sensors like LiDAR, camera images, and GPS. We propose individual modality and distributed fusion-based deep learning (F-DL) architectures that can execute locally as well as at a mobile edge computing center (MEC), with a study on associated tradeoffs. We also formulate and solve an optimization problem that considers practical beam-searching, MEC processing and sensor-to-MEC data delivery latency overheads for determining the output dimensions of the above F-DL architectures. Results from extensive evaluations conducted on publicly available synthetic and home-grown real-world datasets reveal 95% and 96% improvement in beam selection speed over classical RF-only beam sweeping, respectively. F-DL also outperforms the state-of-the-art techniques by 20-22% in predicting top-10 best beam pairs.

Motivation & Objective

• Address the challenge of high beam selection latency in mmWave vehicular communications due to exhaustive RF beam sweeping during short vehicle contact times.
• Leverage non-RF multimodal sensor data (LiDAR, camera, GPS) to predict likely optimal beam pairs and reduce the search space.
• Design distributed deep learning architectures for local and mobile edge computing (MEC) deployment to balance latency, accuracy, and communication overhead.
• Optimize the output dimensions of fusion-based deep learning models by jointly considering beam-searching, MEC processing, and sensor-to-MEC data delivery latencies.
• Improve beam selection performance over classical RF-only sweeping and state-of-the-art methods using real-world and synthetic datasets.

Proposed method

• Collect and preprocess multimodal sensor data: LiDAR for 3D environmental reconstruction, camera images for object detection via a fully convolutional network, and GPS for vehicle positioning.
• Generate a bit map representation of camera images using a sliding window approach with a trained classifier, enabling single-pass inference for low-latency processing.
• Apply object type filtering: map target vehicle type (from safety messages) to legitimate receiver, others to obstacles, refining beam candidate locations.
• Design individual modality and distributed fusion-based deep learning (F-DL) architectures that can be executed locally on vehicles or at MEC centers.
• Formulate and solve an optimization problem to determine optimal output dimensions of F-DL models, minimizing total latency from sensor data delivery to beam selection.
• Use a fusion pipeline that combines GPS, camera, and LiDAR data to restrict beam selection to top-K candidate beam pairs, reducing search space and accelerating convergence.

Experimental results

Research questions

• RQ1 Can multimodal sensor data (LiDAR, camera, GPS) be effectively leveraged to reduce beam selection latency in mmWave vehicular networks?
• RQ2 How does the performance of F-DL models compare to classical RF-only beam sweeping in terms of beam selection speed and accuracy?
• RQ3 What is the optimal trade-off between model complexity, edge processing delay, and communication overhead in distributed beam selection architectures?
• RQ4 To what extent do fusion-based deep learning models improve top-10 beam pair prediction accuracy compared to state-of-the-art methods?
• RQ5 How does the proposed optimization framework for output dimension selection impact end-to-end beam selection latency and throughput?

Key findings

• The proposed F-DL framework achieves 95% and 96% improvement in beam selection speed over classical RF-only beam sweeping on synthetic and real-world datasets, respectively.
• F-DL outperforms state-of-the-art techniques by 20–22% in top-10 beam pair prediction accuracy, measured using the top-K accuracy metric.
• The optimized F-DL model reduces total beam selection latency by jointly minimizing beam-searching, MEC processing, and sensor-to-MEC data delivery overheads.
• The fully convolutional network-based bit map generation enables single-pass inference, reducing image preprocessing delay by eliminating multiple forward passes.
• The fusion pipeline integrating GPS, camera, and LiDAR data significantly improves beam selection accuracy by accurately localizing the target receiver and obstacles.
• Throughput ratio evaluation shows that the proposed method maintains high spectral efficiency, with $ R_T $ values close to 1, indicating minimal performance loss despite reduced search space.