[Paper Review] BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving

Tang, Tao, Dafeng Wei+13 more|arXiv (Cornell University)|Jan 2, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

BEV-CLIP proposes the first multimodal BEV (Bird's-Eye View) retrieval framework that enables zero-shot text-to-scene retrieval in autonomous driving by fusing BEV features from BEVFormer with enriched text embeddings from a fine-tuned LLaMA2 LLM and a knowledge graph. It achieves 87.66% text-to-BEV retrieval accuracy on the NuScenes dataset, demonstrating strong performance on complex and long-tail driving scenarios through a shared cross-modal prompt and multi-task learning.

ABSTRACT

The rapid development of the autonomous driving industry has led to a significant accumulation of autonomous driving data. Consequently, there comes a growing demand for retrieving data to provide specialized optimization. However, directly applying previous image retrieval methods faces several challenges, such as the lack of global feature representation and inadequate text retrieval ability for complex driving scenes. To address these issues, firstly, we propose the BEV-TSR framework which leverages descriptive text as an input to retrieve corresponding scenes in the Bird's Eye View (BEV) space. Then to facilitate complex scene retrieval with extensive text descriptions, we employ a large language model (LLM) to extract the semantic features of the text inputs and incorporate knowledge graph embeddings to enhance the semantic richness of the language embedding. To achieve feature alignment between the BEV feature and language embedding, we propose Shared Cross-modal Embedding with a set of shared learnable embeddings to bridge the gap between these two modalities, and employ a caption generation task to further enhance the alignment. Furthermore, there lack of well-formed retrieval datasets for effective evaluation. To this end, we establish a multi-level retrieval dataset, nuScenes-Retrieval, based on the widely adopted nuScenes dataset. Experimental results on the multi-level nuScenes-Retrieval show that BEV-TSR achieves state-of-the-art performance, e.g., 85.78% and 87.66% top-1 accuracy on scene-to-text and text-to-scene retrieval respectively. Codes and datasets will be available.

Motivation & Objective

• To address the limitations of 2D image-based retrieval in autonomous driving, particularly poor global feature representation and weak text understanding.
• To enable zero-shot retrieval of complex, long-tail driving scenes using descriptive text as input.
• To enhance text representation in autonomous driving by integrating large language models (LLMs) and domain-specific knowledge graphs.
• To develop a unified cross-modal alignment mechanism that fuses BEV and text features effectively for retrieval.
• To improve retrieval robustness on rare and complex scenarios through multi-task learning with caption generation.

Proposed method

• Utilizes BEVFormer to extract global, multi-view BEV features from camera inputs without fine-tuning, enabling unified 3D scene representation.
• Employs a decoder-only LLaMA2 LLM as the text encoder, fine-tuned via LoRA to better understand complex driving scene descriptions.
• Incorporates domain-specific knowledge graph embeddings to enrich semantic representation and improve generalization in autonomous driving contexts.
• Introduces a Shared Cross-Modal Prompt (SCP) to align BEV and text features prior to contrastive learning, enabling effective cross-modal fusion.
• Employs a multi-task learning setup with a caption generation head to improve feature abstraction and generalization.
• Applies contrastive learning to align BEV and text embeddings in a shared latent space, optimizing for retrieval performance.

Experimental results

Research questions

• RQ1 Can BEV features significantly improve global scene representation for text-to-scene retrieval compared to 2D image features?
• RQ2 Can a fine-tuned LLaMA2 LLM outperform standard BERT-based text encoders in understanding complex autonomous driving scene descriptions?
• RQ3 Does integrating knowledge graph embeddings enhance the semantic richness and retrieval accuracy of text features in autonomous driving?
• RQ4 Can a shared cross-modal prompt effectively align BEV and text features for zero-shot retrieval?
• RQ5 Does multi-task learning with caption generation improve the model’s ability to generalize to rare and long-tail driving scenarios?

Key findings

• BEV-CLIP achieves 87.66% text-to-BEV retrieval accuracy on the NuScenes dataset, significantly outperforming baseline methods.
• The use of fine-tuned LLaMA2 with LoRA boosts retrieval performance by approximately 6% and 10% on B2T_R1 and T2B_R1 metrics, respectively, compared to BERT-based baselines.
• Incorporating knowledge graph embeddings improves the semantic richness of text features and enhances model generalization, especially in rare scenarios.
• The Shared Cross-Modal Prompt (SCP) mechanism significantly improves retrieval metrics, even without modifying the backbone encoders.
• The model maintains strong performance (66.37% and 67.90% on BEV-to-text and text-to-BEV rank@1) on the more diverse NuScenes-QA augmented dataset, demonstrating robustness to data distribution shifts.
• Visualization results confirm that BEV-CLIP effectively captures complex, global scene structures that 2D methods fail to represent, especially in long-tail scenarios.