[Paper Review] Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers

Shijie Geng, Peng Gao+6 more|arXiv (Cornell University)|Jul 8, 2020

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a dynamic graph representation learning framework for video dialog using a novel spatio-temporal scene graph (STSGR) and a shuffled Transformer architecture. By modeling intra-frame visual semantics via graph attention and edge convolutions, and aggregating temporal cues across frames, the method achieves state-of-the-art performance on both answer generation and selection tasks in the AVSD benchmark, outperforming prior methods on all metrics including CIDEr (1.272 on DSTC8).

ABSTRACT

Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires an agent to indulge in a question-answer dialog with a human about the audio-visual content. This task thus poses a challenging multi-modal representation learning and reasoning scenario, advancements into which could influence several human-machine interaction applications. To solve this task, we introduce a semantics-controlled multi-modal shuffled Transformer reasoning framework, consisting of a sequence of Transformer modules, each taking a modality as input and producing representations conditioned on the input question. Our proposed Transformer variant uses a shuffling scheme on their multi-head outputs, demonstrating better regularization. To encode fine-grained visual information, we present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing spatio-semantic graph representations for every frame, and an inter-frame aggregation module capturing temporal cues. Our entire pipeline is trained end-to-end. We present experiments on the benchmark AVSD dataset, both on answer generation and selection tasks. Our results demonstrate state-of-the-art performances on all evaluation metrics.

Motivation & Objective

• To address the challenge of multi-modal representation learning and reasoning in audio-visual scene-aware dialog (AVSD), where inputs include video, audio, captions, and dialog history.
• To improve reasoning over complex, multi-modal inputs by capturing fine-grained visual semantics and temporal dependencies in videos.
• To enhance model generalization and reduce overfitting in multi-modal transformers through a novel shuffling mechanism on multi-head attention outputs.
• To develop a hierarchical representation learning pipeline that fuses visual, audio, and linguistic modalities end-to-end for effective dialog generation and selection.

Proposed method

• Proposes a spatio-temporal scene graph representation (STSGR) that constructs frame-level visual graphs using graph attention and edge convolutions to encode object relationships and spatial context.
• Introduces an inter-frame aggregation module that propagates and updates visual representations across temporal neighbors to build compact, dynamic visual memories.
• Employs a multi-modal shuffled Transformer, where outputs from multi-head attention heads are randomly shuffled before fusion, improving regularization and generalization.
• Fuses audio-visual memories with dialog history, captions, and question embeddings through sequential Transformer decoders, conditioned on the input question.
• Uses end-to-end training to jointly optimize visual graph learning, modality fusion, and answer generation or selection.
• Leverages temporal alignment between audio features and video frames to ensure modality consistency in representation learning.

Experimental results

Research questions

• RQ1 Can a dynamic, graph-based visual representation improve reasoning in video dialog systems compared to holistic video features?
• RQ2 How does the proposed shuffled Transformer architecture enhance generalization and reduce overfitting in multi-modal sequence modeling?
• RQ3 To what extent does incorporating intra-frame and inter-frame reasoning improve performance on complex AVSD tasks?
• RQ4 Does the integration of scene graphs with audio-visual and linguistic modalities lead to better answer generation and selection than prior methods?
• RQ5 How do the model's performance and robustness vary when key modalities (e.g., caption or dialog history) are removed?

Key findings

• The proposed STSGR model achieves state-of-the-art performance on the AVSD benchmark, with a CIDEr score of 1.272 on DSTC8 and 1.249 on DSTC7, surpassing prior methods including MTN and Multi-modal Attention.
• On the answer selection task, the model achieves a mean retrieval rank of 4.08 (full model) and 5.91 (no caption), significantly outperforming the baseline of 7.41 and 6.54, respectively.
• The ablation study confirms that incorporating audio into the STSGR representation improves performance, reducing mean retrieval rank from 4.67 to 4.08 when full modality is used.
• The STSGR representation alone outperforms I3D and VGG features on the generation task, with a CIDEr score of 1.265 using scene graphs versus 1.249 for MTN using I3D.
• Qualitative results show the model effectively tracks objects across frames and answers spatial-temporal questions with high confidence, even in cluttered scenes.
• The shuffled Transformer variant reduces overfitting and improves generalization, as evidenced by consistent performance gains across all evaluation metrics and ablation settings.