[Paper Review] Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation

Yan Xu, Zhihao Yuan+5 more|arXiv (Cornell University)|Dec 22, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces CLEVR3D, a large-scale VQA-3D dataset with 171K questions across 8,771 real and augmented 3D scenes, generated via a 3D scene graph-based question engine and compositional scene manipulation to enable common-sense-independent reasoning. The proposed TransVQA3D framework jointly learns VQA-3D and scene graph generation, achieving state-of-the-art performance by leveraging cross-modal attention and scene graph-aware supervision.

ABSTRACT

Visual Question Answering on 3D Point Cloud (VQA-3D) is an emerging yet challenging field that aims at answering various types of textual questions given an entire point cloud scene. To tackle this problem, we propose the CLEVR3D, a large-scale VQA-3D dataset consisting of 171K questions from 8,771 3D scenes. Specifically, we develop a question engine leveraging 3D scene graph structures to generate diverse reasoning questions, covering the questions of objects' attributes (i.e., size, color, and material) and their spatial relationships. Through such a manner, we initially generated 44K questions from 1,333 real-world scenes. Moreover, a more challenging setup is proposed to remove the confounding bias and adjust the context from a common-sense layout. Such a setup requires the network to achieve comprehensive visual understanding when the 3D scene is different from the general co-occurrence context (e.g., chairs always exist with tables). To this end, we further introduce the compositional scene manipulation strategy and generate 127K questions from 7,438 augmented 3D scenes, which can improve VQA-3D models for real-world comprehension. Built upon the proposed dataset, we baseline several VQA-3D models, where experimental results verify that the CLEVR3D can significantly boost other 3D scene understanding tasks. Our code and dataset will be made publicly available at https://github.com/yanx27/CLEVR3D.

Motivation & Objective

• To address the lack of large-scale, diverse, and bias-free 3D VQA datasets that support complex reasoning beyond object recognition.
• To mitigate spurious correlations in VQA-3D models by introducing a common-sense-independent setup through compositional scene manipulation.
• To enable comprehensive visual understanding by generating questions on object attributes, spatial relationships, and reasoning types such as counting and comparison.
• To develop a unified framework that jointly optimizes VQA-3D and 3D scene graph generation for improved reasoning and robustness.
• To provide a benchmark that enhances 3D scene understanding across multiple downstream tasks.

Proposed method

• A question engine is designed based on 3D semantic scene graphs to generate 44K diverse reasoning questions from 1,333 real-world 3D scenes, covering attributes (color, size, material) and spatial relationships.
• Compositional Scene Manipulation (CSM) is introduced to alter scene graphs by reassigning object relationships and attributes, generating 127K questions from 7,438 augmented scenes to eliminate common-sense priors.
• The TransVQA3D framework employs a Cross-Modal Transformer (CMT) to align point cloud features with textual questions and uses a Scene Graph Attention Adapter (SGAA) to inject explicit relationship supervision.
• The model jointly optimizes for VQA-3D and 3D scene graph generation, with separate head branches for object classification and predicate prediction, using top-k recall and accuracy metrics.
• The SGAA module incorporates twin attention (TA) for node-level attention and cross-modal attention (CMA) to enhance feature interaction between visual and language modalities.
• Positional encoding is applied in the CMT to preserve spatial and sequential structure, significantly improving performance as shown in ablation studies.

Experimental results

Research questions

• RQ1 Can a 3D scene graph-based question engine generate diverse, reasoning-intensive questions covering attributes and spatial relationships in real-world 3D scenes?
• RQ2 How effective is compositional scene manipulation in reducing model reliance on spurious common-sense biases in VQA-3D?
• RQ3 To what extent does joint learning of VQA-3D and 3D scene graph generation improve reasoning performance and robustness?
• RQ4 What is the contribution of explicit relationship supervision via the SGAA module in enhancing VQA-3D and scene graph prediction accuracy?
• RQ5 How do architectural components like positional encoding and cross-modal attention influence the performance of the CMT in 3D VQA?

Key findings

• The CLEVR3D dataset contains 171,000 questions from 8,771 scenes, with 44,000 from real-world scenes and 127,000 from augmented, common-sense-independent scenes via compositional manipulation.
• TransVQA3D achieves a 39.7% overall accuracy on VQA-3D when trained only on the VQA task, which improves to 42.7% with object classification supervision and to 44.7% with full scene graph supervision via the SGAA module.
• The ablation study shows that removing the language branch degrades scene graph prediction by approximately 5%, demonstrating mutual enhancement between VQA-3D and scene graph reasoning.
• The use of positional encoding in the CMT reduces overall accuracy drop by 3% and ClassAvg drop by 6% compared to models without it, proving its critical role in modeling cross-modal interactions.
• The twin attention (TA) component in the SGAA module significantly improves both VQA-3D and scene graph prediction, while cross-modal attention (CMA) mainly benefits VQA-3D with a 2% gain.
• The model achieves state-of-the-art performance on scene graph analysis, especially in predicate (relationship) prediction, with notable improvements in top-k recall metrics.