[Paper Review] MTGAT: Multimodal Temporal Graph Attention Networks for Unaligned Human Multimodal Language Sequences.

Jianing Yang, Yongxin Wang+6 more|arXiv (Cornell University)|Oct 22, 2020

Sentiment Analysis and Opinion Mining9 citations

TL;DR

MTGAT proposes a novel graph-based neural network that models unaligned multimodal human language sequences by converting them into heterogeneous temporal graphs, using Multimodal Temporal Graph Attention with dynamic pruning to focus on salient interactions. It achieves state-of-the-art performance on IEMOCAP and CMU-MOSI with reduced computational cost.

ABSTRACT

Human communication is multimodal in nature; it is through multiple modalities, i.e., language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Multimodal Temporal Graph Attention Networks (MTGAT). MTGAT is an interpretable graph-based neural model that provides a suitable framework for analyzing this type of multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions between different modalities through time. Then, a novel graph operation, called Multimodal Temporal Graph Attention, along with a dynamic pruning and read-out technique is designed to efficiently process this multimodal temporal graph. By learning to focus only on the important interactions within the graph, our MTGAT is able to achieve state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks including IEMOCAP and CMU-MOSI, while utilizing significantly fewer computations.

Motivation & Objective

• To address the challenge of modeling complex, unaligned multimodal interactions in human communication, such as language, voice, and facial expressions.
• To develop a framework that captures rich temporal and relational dynamics across multiple modalities in sequential data.
• To design a graph-based architecture that enables interpretable attention over multimodal interactions while reducing computational overhead.

Proposed method

• Convert unaligned multimodal sequences into a heterogeneous graph with nodes representing modalities and edges encoding temporal and cross-modal relationships.
• Introduce Multimodal Temporal Graph Attention, a novel graph operation that learns dynamic attention weights over graph edges to emphasize important interactions.
• Apply a dynamic pruning mechanism to remove low-importance edges during inference, reducing computational complexity without sacrificing performance.
• Use a read-out mechanism to aggregate graph representations into a unified multimodal embedding for downstream tasks.
• Train the model end-to-end using a loss function tailored for sentiment analysis and emotion recognition.
• Leverage attention mechanisms to interpret which modalities and time steps contribute most to predictions.

Experimental results

Research questions

• RQ1 How can unaligned multimodal human sequences be effectively modeled as a graph to preserve temporal and cross-modal dependencies?
• RQ2 What graph-based attention mechanism enables efficient and interpretable focus on salient interactions in multimodal data?
• RQ3 To what extent can dynamic pruning reduce computational cost while maintaining high performance in multimodal understanding?
• RQ4 Can the proposed model outperform existing methods on multimodal sentiment and emotion recognition benchmarks?

Key findings

• MTGAT achieves state-of-the-art performance on the IEMOCAP benchmark for multimodal sentiment analysis.
• The model attains new state-of-the-art results on the CMU-MOSI dataset for multimodal emotion recognition.
• Despite its high performance, MTGAT uses significantly fewer computations than prior models due to dynamic pruning.
• The attention mechanism in MTGAT provides interpretable insights into which modalities and time intervals are most influential in predictions.
• The graph-based representation effectively captures complex, multi-relational interactions across unaligned modalities.
• The dynamic pruning strategy reduces computational load while preserving predictive accuracy.