[Paper Review] Location-aware Graph Convolutional Networks for Video Question Answering

Deng Huang, Peihao Chen+4 more|arXiv (Cornell University)|Aug 7, 2020

Multimodal Machine Learning Applications39 references4 citations

TL;DR

This paper proposes Location-aware Graph Convolutional Networks (L-GCN) for video question answering by modeling object interactions through a graph structure enriched with spatial and temporal location features. By leveraging graph convolution to capture direct relationships among detected objects while filtering out background clutter, L-GCN achieves state-of-the-art performance on TGIF-QA, YouTube2Text-QA, and MSVD-QA benchmarks, with significant gains on temporal reasoning tasks.

ABSTRACT

We addressed the challenging task of video question answering, which requires machines to answer questions about videos in a natural language form. Previous state-of-the-art methods attempt to apply spatio-temporal attention mechanism on video frame features without explicitly modeling the location and relations among object interaction occurred in videos. However, the relations between object interaction and their location information are very critical for both action recognition and question reasoning. In this work, we propose to represent the contents in the video as a location-aware graph by incorporating the location information of an object into the graph construction. Here, each node is associated with an object represented by its appearance and location features. Based on the constructed graph, we propose to use graph convolution to infer both the category and temporal locations of an action. As the graph is built on objects, our method is able to focus on the foreground action contents for better video question answering. Lastly, we leverage an attention mechanism to combine the output of graph convolution and encoded question features for final answer reasoning. Extensive experiments demonstrate the effectiveness of the proposed methods. Specifically, our method significantly outperforms state-of-the-art methods on TGIF-QA, Youtube2Text-QA, and MSVD-QA datasets. Code and pre-trained models are publicly available at: https://github.com/SunDoge/L-GCN

Motivation & Objective

• To address the challenge of video question answering by explicitly modeling object interactions and their spatiotemporal locations, which are critical for accurate reasoning.
• To overcome limitations of existing methods that rely on spatio-temporal attention or RNNs, which are sensitive to background content and struggle with non-adjacent object interactions.
• To improve performance on temporal reasoning tasks such as action transition and counting by incorporating both spatial and temporal location cues into the graph structure.
• To develop a graph-based framework that enables direct, non-recurrent interaction between all objects, enhancing modeling of complex video dynamics.

Proposed method

• Objects are detected in each video frame using an off-the-shelf detector and represented as nodes in a fully connected graph.
• Each node is embedded with both appearance features and 2D spatial + temporal location coordinates, making the graph location-aware.
• Graph convolutional networks (GCNs) are applied to propagate messages across nodes, enabling direct interaction between all objects regardless of frame proximity.
• The GCN output is fused with encoded question features via a visual-question interaction module to predict the final answer.
• The model uses a two-layer GCN by default, with 5 detected objects per frame, based on ablation studies showing optimal performance.
• Location features are incorporated as additional input to each node, with ablation showing that both spatial and temporal locations are complementary and essential.

Experimental results

Research questions

• RQ1 Can modeling object interactions through a location-aware graph improve video question answering performance compared to attention-based or RNN-based methods?
• RQ2 How does incorporating both spatial and temporal location information affect reasoning on temporal queries such as action order or repetition count?
• RQ3 To what extent does a graph-based approach that enables direct object interaction outperform recurrent or sequential modeling in video QA?
• RQ4 What is the optimal depth of GCN layers and number of detected objects per frame for balancing performance and efficiency in video QA?

Key findings

• L-GCN achieves state-of-the-art performance on TGIF-QA, with a 74.32% accuracy on the action prediction task, outperforming the baseline by 3.74 percentage points.
• On the transition task, L-GCN improves from 79.59% (baseline) to 81.13%, demonstrating enhanced reasoning over action order.
• The model shows significant gains on the count task (from 4.33 to 3.95), indicating better understanding of action repetition and temporal patterns.
• Ablation studies confirm that incorporating both spatial and temporal location features yields the highest performance, with individual use of either type resulting in lower accuracy.
• Using two GCN layers yields the best performance, while 5 detected objects per frame provide the optimal trade-off between coverage and noise.
• Qualitative analysis confirms that the model effectively detects salient, question-relevant objects and captures semantic relationships across frames, such as between a man and a motorcycle in a dynamic scene.