[Paper Review] GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models

Haicheng Liao, Huanming Shen+5 more|arXiv (Cornell University)|Dec 6, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes CAVG, a novel encoder-decoder framework that enhances visual grounding in autonomous driving by integrating GPT-4-powered emotion modeling, multi-head cross-modal attention, and a Region-Specific Dynamic (RSD) layer. It achieves state-of-the-art accuracy on the Talk2Car dataset, even with 50–75% of the training data, and demonstrates robust performance across challenging real-world scenarios including low-light, inclement weather, and ambiguous commands.

ABSTRACT

In the field of autonomous vehicles (AVs), accurately discerning commander intent and executing linguistic commands within a visual context presents a significant challenge. This paper introduces a sophisticated encoder-decoder framework, developed to address visual grounding in AVs.Our Context-Aware Visual Grounding (CAVG) model is an advanced system that integrates five core encoders-Text, Image, Context, and Cross-Modal-with a Multimodal decoder. This integration enables the CAVG model to adeptly capture contextual semantics and to learn human emotional features, augmented by state-of-the-art Large Language Models (LLMs) including GPT-4. The architecture of CAVG is reinforced by the implementation of multi-head cross-modal attention mechanisms and a Region-Specific Dynamic (RSD) layer for attention modulation. This architectural design enables the model to efficiently process and interpret a range of cross-modal inputs, yielding a comprehensive understanding of the correlation between verbal commands and corresponding visual scenes. Empirical evaluations on the Talk2Car dataset, a real-world benchmark, demonstrate that CAVG establishes new standards in prediction accuracy and operational efficiency. Notably, the model exhibits exceptional performance even with limited training data, ranging from 50% to 75% of the full dataset. This feature highlights its effectiveness and potential for deployment in practical AV applications. Moreover, CAVG has shown remarkable robustness and adaptability in challenging scenarios, including long-text command interpretation, low-light conditions, ambiguous command contexts, inclement weather conditions, and densely populated urban environments. The code for the proposed model is available at our Github.

Motivation & Objective

• Address the challenge of accurately grounding natural language commands in complex, dynamic visual scenes for autonomous vehicles (AVs).
• Improve human-AV interaction by modeling emotional context in linguistic commands, which traditional models overlook.
• Enhance visual grounding robustness under real-world conditions such as low-light, bad weather, and dense urban environments.
• Achieve high prediction accuracy and inference efficiency with minimal training data, enabling practical deployment in data-scarce scenarios.
• Develop a unified multimodal framework that integrates text, vision, context, emotion, and cross-modal features for holistic understanding.

Proposed method

• Employ a five-encoder architecture: Text (BERT-based), Emotion (GPT-4 fine-tuned for sentiment and intent), Vision (ViT and BLIP for scene understanding), Context (temporal and spatial context modeling), and Cross-Modal (multi-head attention for cross-modal fusion).
• Integrate a Multimodal Decoder with a Region-Specific Dynamic (RSD) layer to modulate attention and enhance localization precision.
• Use multi-head cross-modal attention to dynamically align linguistic tokens with relevant visual regions across modalities.
• Leverage GPT-4 to extract and classify emotional cues from commands, enriching intent understanding beyond semantics.
• Train the end-to-end CAVG framework on the Talk2Car dataset using contrastive learning and masked language modeling objectives.
• Optimize for real-time inference by balancing model depth and attention efficiency, ensuring low-latency responses.

Figure 1: Illustration of Regions Identified by an AV based on a Raw Image and a Natural Language Command. The blue bounding box represents the ground truth. The red and yellow bounding boxes correspond to the prediction results from CAVG with emotion categorization and without emotion categorizatio

Experimental results

Research questions

• RQ1 Can integrating GPT-4-based emotion modeling improve the accuracy and contextual relevance of visual grounding in autonomous driving?
• RQ2 How does the CAVG framework perform under low-data regimes (50–75% of full dataset) compared to existing state-of-the-art models?
• RQ3 To what extent does the RSD layer and multi-head cross-modal attention enhance localization precision in complex visual scenes?
• RQ4 How does the model handle ambiguous, long-text, or emotionally charged commands in real-world driving scenarios?
• RQ5 What is the user-perceived quality of CAVG in terms of response accuracy, inference speed, and overall user experience compared to existing models?

Key findings

• CAVG achieves state-of-the-art prediction accuracy on the Talk2Car benchmark, outperforming existing models including Stacked VL-BERT, CMSVG, and AttnGrounder.
• The model maintains high performance with only 50–75% of the full training data, demonstrating strong generalization and data efficiency.
• In user studies, CAVG received the highest average ratings—5.2% to 10.6% higher than competitors—across response accuracy, inference time, and user experience.
• CAVG was selected as the favorite model by the largest proportion of survey participants, indicating superior user satisfaction and practical usability.
• The model exhibits robust performance in challenging conditions, including low-light night scenes, rainy weather, and densely populated urban streets.
• The integration of GPT-4 for emotion classification significantly improves intent understanding, especially in commands with urgent, directive, or emotional tone.

Figure 2: Schematic of the Model Architecture. The Text Encoder and the Emotion Encoder generate a text vector and an emotion vector, respectively, from the given command, while the Vision Encoder divides the input image into $N$ RoIs, each represented by a vision vector. These vectors are contextua