[Paper Review] Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Zekang Li, Zongjia Li+4 more|arXiv (Cornell University)|Feb 1, 2020

Multimodal Machine Learning Applications24 references22 citations

TL;DR

This paper proposes a universal multimodal transformer that fine-tunes a pre-trained GPT-2 model with multi-task learning to jointly encode video, audio, captions, and dialogue history for scene-aware dialog generation. By integrating text, video, and audio features into a unified sequence and training on response language modeling, video-audio sequence modeling, and caption language modeling, the method achieves state-of-the-art performance on the DSTC8-AVSD benchmark, outperforming prior models in both automatic and human evaluations.

ABSTRACT

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

Motivation & Objective

• To address the challenge of generating fluent, contextually grounded responses in video-audio dialogues by integrating multiple modalities effectively.
• To improve joint representation learning across text, video, and audio for better dialogue generation.
• To extend pre-trained language models like GPT-2 to multimodal dialogue generation with self-supervised multi-task learning.
• To achieve state-of-the-art results in both automatic and human evaluations on the AVSD task.

Proposed method

• The model uses a unified sequence input combining video, audio, caption, and dialogue history features, each prefixed with a unique segment token (e.g., [video], [caption], [user1]).
• It initializes with a pre-trained GPT-2 model to leverage strong language generation capacity.
• Multi-task learning is employed with three objectives: Response Language Modeling (RLM), Video-Audio Sequence Modeling (VASM), and Caption Language Modeling (CLM).
• The model is fine-tuned end-to-end using masked language modeling on all modalities to learn joint representations.
• Video and audio features are processed via a 3D CNN and audio embeddings are fused with visual features before input to the transformer.
• Decoding uses beam search, which is shown to yield the best automatic metrics on the AVSD task.

Experimental results

Research questions

• RQ1 Can a pre-trained language model like GPT-2 be effectively adapted to multimodal dialogue generation with video and audio inputs?
• RQ2 How does multi-task learning across response generation, video-audio sequence modeling, and caption reconstruction improve joint representation learning?
• RQ3 Does joint encoding of text, video, and audio outperform separate modality encoders with attention fusion?
• RQ4 How does the model perform when only dialogue history and video-audio are available, without captions or summaries?
• RQ5 What decoding strategy yields the best performance in a grounded, scene-aware dialogue setting?

Key findings

• The model achieves the highest human evaluation score (3.934) among all DSTC8-AVSD submissions, very close to the human reference score of 4.000.
• In the text + video setting, the model improves BLEU-4 by 0.056 and CIDEr by 0.216 over the previous SOTA model (MTN).
• In the text-only setting, the model outperforms the DSTC7-AVSD winning system, improving BLEU-4 by 0.069 and CIDEr by 0.185.
• The use of video-audio sequence modeling (VASM) improves BLEU-4 by 0.011 and CIDEr by 0.026 in the full input setting.
• Beam search decoding yields the best automatic metrics, outperforming greedy search and nucleus sampling.
• The model performs poorly in the text + video w/o caption/summary setting when using multi-task learning, indicating limitations in video representation learning without auxiliary supervision.