[Paper Review] Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering

Zenan Xu, Wanjun Zhong+3 more|arXiv (Cornell University)|May 13, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes SCAN, a cross-modality-aware syntactic hypergraph convolution network that models multi-level semantic compositions in video question answering by constructing a syntactic hypergraph from dependency trees and using optimal transport for sparse, accurate cross-modal alignment. The method outperforms strong baselines on three VideoQA benchmarks by explicitly capturing compositional semantics and filtering noise.

ABSTRACT

A key challenge in video question answering is how to realize the cross-modal semantic alignment between textual concepts and corresponding visual objects. Existing methods mostly seek to align the word representations with the video regions. However, word representations are often not able to convey a complete description of textual concepts, which are in general described by the compositions of certain words. To address this issue, we propose to first build a syntactic dependency tree for each question with an off-the-shelf tool and use it to guide the extraction of meaningful word compositions. Based on the extracted compositions, a hypergraph is further built by viewing the words as nodes and the compositions as hyperedges. Hypergraph convolutional networks (HCN) are then employed to learn the initial representations of word compositions. Afterwards, an optimal transport based method is proposed to perform cross-modal semantic alignment for the textual and visual semantic space. To reflect the cross-modal influences, the cross-modal information is incorporated into the initial representations, leading to a model named cross-modality-aware syntactic HCN. Experimental results on three benchmarks show that our method outperforms all strong baselines. Further analyses demonstrate the effectiveness of each component, and show that our model is good at modeling different levels of semantic compositions and filtering out irrelevant information.

Motivation & Objective

• To address the limitation of word-level alignment in VideoQA, which fails to capture complete textual concepts like 'girl in green' or 'girl sitting on'.
• To model multi-level semantic compositions (e.g., word, phrase, sentence) in natural language questions using syntactic dependency structures.
• To improve cross-modal alignment between textual concepts and visual objects by leveraging hypergraph convolution and optimal transport.
• To enhance representation learning by incorporating cross-modal information into initial semantic compositions.
• To demonstrate the effectiveness of syntactic hypergraph modeling in filtering out irrelevant visual and textual signals.

Proposed method

• Construct a syntactic dependency tree for each question using an off-the-shelf NLP tool to identify meaningful word compositions.
• Build a syntactic hypergraph where words are nodes and syntactic subtrees (compositions) are hyperedges.
• Apply hypergraph convolutional networks (HCN) to learn initial representations of these semantic compositions.
• Introduce a cross-modality-aware mechanism that updates textual and visual representations using cross-modal information.
• Employ optimal transport (OT) to compute sparse, accurate cross-modal alignments between textual compositions and visual features.
• Use a hierarchical bottom-up approach to build the syntactic hypergraph, enabling modeling of semantic compositions at different levels.

Experimental results

Research questions

• RQ1 Can modeling multi-level semantic compositions in questions improve performance in video question answering?
• RQ2 Does using syntactic dependency trees to extract meaningful word compositions lead to better alignment with visual regions?
• RQ3 Can hypergraph convolutional networks effectively learn representations of compositional textual concepts in VideoQA?
• RQ4 Does optimal transport-based alignment produce sparser and more accurate cross-modal matching than dot-product attention?
• RQ5 How does incorporating cross-modal information into initial representations affect model robustness and performance?

Key findings

• SCAN achieves state-of-the-art performance on three VideoQA benchmarks: 40.3% on MSVD-QA and 37.1% on MSRVTT-QA, outperforming strong baselines.
• Ablation studies show that removing the syntactic hypergraph or the cross-modality-aware module significantly degrades performance, confirming their importance.
• The optimal number of HCN layers varies by task (e.g., l=2 for Transition and FrameQA, l=3 for Action), indicating task-specific needs for compositional reasoning depth.
• Qualitative analysis confirms that SCAN aligns better with relevant video frames at phrase and sentence levels, such as 'gets into the net' focusing on goal-scoring actions.
• The optimal transport-based alignment produces sparser attention matrices than dot-product methods, leading to more focused and accurate cross-modal matching.
• Even when compared to BERT-large, SCAN achieves superior results, demonstrating that explicit syntactic composition modeling adds value beyond pre-trained contextual embeddings.