[论文解读] SkipcrossNets: Adaptive Skip-cross Fusion for Road Detection

Yan Gong, Xinyu Zhang+8 more|arXiv (Cornell University)|Aug 24, 2023

Advanced Neural Network ApplicationsComputer Science被引用 3

一句话总结

该论文提出了一种新型自适应跳过-交叉融合架构SkipcrossNets，实现了LiDAR点云与相机图像在道路检测中的动态、多层级融合。通过在所有网络层之间建立密集的跨模态跳过连接，该方法自适应地选择最优融合点，在KITTI数据集上达到96.85%的MaxF，在A2D2数据集上达到84.84%的F1，同时保持了2.33 MB的小型模型尺寸和68.24 FPS的推理速度，适用于实时部署。

ABSTRACT

Multi-modal fusion is increasingly being used for autonomous driving tasks, as different modalities provide unique information for feature extraction. However, the existing two-stream networks are only fused at a specific network layer, which requires a lot of manual attempts to set up. As the CNN goes deeper, the two modal features become more and more advanced and abstract, and the fusion occurs at the feature level with a large gap, which can easily hurt the performance. To reduce the loss of height and depth information during the process of projecting point clouds into 2D space, we utilize calibration parameters to project the point cloud into Altitude Difference Images (ADIs), which exhibit more distinct road features. In this study, we propose a novel fusion architecture called Skip-cross Networks (SkipcrossNets), which combine adaptively ADIs and camera images without being bound to a certain fusion epoch. Specifically, skip-cross fusion strategy connects each layer to each layer in a feed-forward manner, and for each layer, the feature maps of all previous layers are used as input and its own feature maps are used as input to all subsequent layers for the other modality, enhancing feature propagation and multi-modal features fusion. This strategy facilitates selection of the most similar feature layers from two modalities, enhancing feature reuse and providing complementary effects for sparse point cloud features. The advantages of skip-cross fusion strategy is demonstrated through application to the KITTI and A2D2 datasets, achieving a MaxF score of 96.85% on KITTI and an F1 score of 84.84% on A2D2. The model parameters require only 2.33 MB of memory at a speed of 68.24 FPS, which can be viable for mobile terminals and embedded devices.

研究动机与目标

• 解决两流网络在多模态道路检测中融合时机不理想的问题。
• 减少对深度网络中融合层选择的手动超参数调优依赖。
• 增强稀疏LiDAR特征与高分辨率相机特征之间的特征传播与互补性。
• 开发一种轻量化、具备实时处理能力的模型，适用于嵌入式和移动自动驾驶系统。
• 通过在模态间实现自适应、跨层级的融合，提升稀疏点云数据上的性能。

提出的方法

• 提出一种跳过-交叉融合机制，以前馈方式将每一层的特征图与另一模态的所有后续层连接。
• 引入跳过-交叉模块，通过学习各特征层级上的最优融合权重，实现自适应、动态融合。
• 采用多尺度编码器-解码器架构，结合残差块（ResNet34主干网络），以保留空间细节并增强特征学习能力。
• 在编码器与解码器之间使用跳跃连接，以保留低级特征并改善分辨率恢复。
• 应用自适应距离图像（ADIs）将3D LiDAR点云投影至2D空间，提升稀疏数据的特征表示能力。
• 设计了一种轻量化网络结构，仅含2.33 MB参数，实现68.24 FPS的高速推理。

实验结果

研究问题

• RQ1 如何在多个特征层级上实现多模态融合的自适应性，以避免手动选择融合时机？
• RQ2 跨层级、密集的跳跃连接对LiDAR与相机模态之间特征互补性有何影响？
• RQ3 自适应融合是否能在不增加模型复杂度的前提下提升稀疏LiDAR数据上的性能？
• RQ4 与早期、中期和晚期融合策略相比，所提出的跳过-交叉融合在准确率和效率方面表现如何？
• RQ5 该融合机制在不同网络架构和数据集上的泛化能力如何？

主要发现

• SkipcrossNets在KITTI数据集上取得了96.85%的MaxF分数，优于现有方法。
• 在A2D2数据集上，模型达到了84.84%的F1分数，表明其在多样化道路场景中具有强大的泛化能力。
• 模型实现了68.24 FPS的推理速度，适合在移动和嵌入式平台实现实时部署。
• 网络参数量仅为2.33 MB，显著小于同类SOTA模型。
• 消融实验表明，与基线融合策略相比，跳过-交叉融合使KITTI的MaxF提升了1.11%，A2D2的F1提升了1.18%。
• 添加ResNet34主干网络和跳跃连接进一步提升了性能，使MaxF提升1.32%，F1提升1.27%。