[Paper Review] NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario

Tianwen Qian, Jingjing Chen+3 more|arXiv (Cornell University)|May 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces NuScenes-QA, the first large-scale multi-modal visual question answering benchmark for autonomous driving, leveraging 34K scenes and 460K question-answer pairs derived from nuScenes using scene graphs and handcrafted templates. The benchmark integrates multi-frame, multi-modal (image and LiDAR) data and evaluates models on complex spatial and dynamic reasoning, with ablation studies showing that ground-truth bounding boxes significantly boost performance, while detected boxes can degrade it due to detection noise.

ABSTRACT

We introduce a novel visual question answering (VQA) task in the context of autonomous driving, aiming to answer natural language questions based on street-view clues. Compared to traditional VQA tasks, VQA in autonomous driving scenario presents more challenges. Firstly, the raw visual data are multi-modal, including images and point clouds captured by camera and LiDAR, respectively. Secondly, the data are multi-frame due to the continuous, real-time acquisition. Thirdly, the outdoor scenes exhibit both moving foreground and static background. Existing VQA benchmarks fail to adequately address these complexities. To bridge this gap, we propose NuScenes-QA, the first benchmark for VQA in the autonomous driving scenario, encompassing 34K visual scenes and 460K question-answer pairs. Specifically, we leverage existing 3D detection annotations to generate scene graphs and design question templates manually. Subsequently, the question-answer pairs are generated programmatically based on these templates. Comprehensive statistics prove that our NuScenes-QA is a balanced large-scale benchmark with diverse question formats. Built upon it, we develop a series of baselines that employ advanced 3D detection and VQA techniques. Our extensive experiments highlight the challenges posed by this new task. Codes and dataset are available at https://github.com/qiantianwen/NuScenes-QA.

Motivation & Objective

• To address the lack of large-scale, multi-modal, multi-frame VQA benchmarks tailored for autonomous driving scenarios.
• To bridge the gap between existing VQA benchmarks and the complex, real-world demands of street-level perception in autonomous vehicles.
• To develop a scalable, programmatic pipeline for generating diverse, balanced, and realistic VQA data from 3D detection annotations.
• To establish a foundation for evaluating and advancing multi-modal, multi-frame VQA models in outdoor, dynamic driving environments.

Proposed method

• Construct scene graphs from 3D bounding box annotations in nuScenes, representing objects as nodes and spatial relationships as edges.
• Design a set of question templates covering existence, counting, comparison, and attribute-based queries using parameters for attributes, objects, and relationships.
• Programmatically generate question-answer pairs by sampling parameters from scene graphs and inferring correct answers using geometric and spatial reasoning.
• Utilize a multi-modal fusion framework combining 2D images and 3D point clouds, with BEV (Bird's Eye View) feature extraction and rotated RoI pooling for spatially accurate feature cropping.
• Apply mean pooling over cropped BEV features to preserve structural information, outperforming max pooling in spatial reasoning tasks.
• Integrate 3D detection models (e.g., CenterPoint) and VQA heads (e.g., MCAN) to build strong baselines, evaluating the impact of ground-truth vs. predicted bounding boxes.

Experimental results

Research questions

• RQ1 How does the performance of VQA models vary when using ground-truth versus detected 3D bounding boxes in autonomous driving scenarios?
• RQ2 To what extent does multi-modal fusion of camera and LiDAR data improve VQA accuracy on complex street scenes?
• RQ3 How do different feature cropping and pooling strategies affect model performance in BEV-based VQA systems?
• RQ4 What is the impact of scene graph-based, template-driven data generation on the diversity and balance of VQA question formats?
• RQ5 Can existing 2D and 3D VQA models generalize to multi-frame, outdoor, dynamic driving scenes, or are new architectures required?

Key findings

• The use of ground-truth 3D bounding boxes improves VQA accuracy from 70.8% to 84.3%, demonstrating their critical role in spatial reasoning.
• Detected 3D bounding boxes slightly reduce performance by 0.6%, likely due to noise in current 3D detection models.
• Rotated box-based feature cropping outperforms circumscribed rectangle cropping by 0.8% in overall accuracy, especially benefiting elongated objects like buses and trucks.
• Mean pooling of cropped BEV features yields 0.6% higher accuracy than max pooling, indicating better preservation of structural features.
• The proposed NuScenes-QA benchmark contains 34K scenes and 460K question-answer pairs, with 377K for training and 83K for testing, ensuring broad coverage of question types.
• The benchmark reveals significant challenges in multi-modal, multi-frame VQA, highlighting the need for improved perception and reasoning models in autonomous driving.