[Paper Review] Multimodal Dialogue State Tracking By QA Approach with Data Augmentation

Xiangyang Mou, Brandyn Sigouin+2 more|arXiv (Cornell University)|Jul 20, 2020

Multimodal Machine Learning Applications21 references5 citations

TL;DR

This paper proposes a multimodal open-domain QA framework for Audio-Video Scene-Aware Dialog (AVSD) using a sequence-to-sequence model with multimodal fusion and attention, trained via teacher forcing. It introduces a novel data augmentation technique that shuffles dialog history QA pairs to improve generalization, achieving significant gains over baseline models on the DSTC7-AVSD dataset.

ABSTRACT

Recently, a more challenging state tracking task, Audio-Video Scene-Aware Dialogue (AVSD), is catching an increasing amount of attention among researchers. Different from purely text-based dialogue state tracking, the dialogue in AVSD contains a sequence of question-answer pairs about a video and the final answer to the given question requires additional understanding of the video. This paper interprets the AVSD task from an open-domain Question Answering (QA) point of view and proposes a multimodal open-domain QA system to deal with the problem. The proposed QA system uses common encoder-decoder framework with multimodal fusion and attention. Teacher forcing is applied to train a natural language generator. We also propose a new data augmentation approach specifically under QA assumption. Our experiments show that our model and techniques bring significant improvements over the baseline model on the DSTC7-AVSD dataset and demonstrate the potentials of our data augmentation techniques.

Motivation & Objective

• To address the challenge of multimodal dialogue state tracking in AVSD by reframing it as an open-domain question answering problem.
• To improve natural language generation in multimodal dialogues by modeling temporal dependencies in generated responses.
• To investigate whether dialog history content matters more than its order for question answering in multimodal settings.
• To develop a data augmentation strategy that increases training data diversity by shuffling QA pairs within dialog histories.
• To evaluate the effectiveness of teacher forcing and pooling mechanisms in improving response generation quality.

Proposed method

• Adopts an encoder-decoder architecture with multimodal fusion, combining visual, audio, and textual encodings for joint representation learning.
• Employs bidirectional GRUs (BiGRUs) as the sequence encoder for dialog history, with max pooling to extract dominant features across time steps.
• Applies teacher forcing during training to enforce autoregressive generation, where ground-truth tokens are used as inputs for next-step prediction.
• Introduces a data augmentation technique that shuffles the order of QA pairs in dialog history to increase training data diversity while preserving semantic content.
• Uses scheduled sampling as a baseline comparison but finds it less effective than strict teacher forcing due to noisy predictions during training.
• Employs cross-attention mechanisms to align question representations with multimodal context representations for improved answer generation.

Experimental results

Research questions

• RQ1 Does modeling dialog state tracking as a QA task improve performance on multimodal dialog understanding?
• RQ2 Is the order of QA pairs in dialog history critical for accurate answer generation, or is content alone sufficient?
• RQ3 How effective is teacher forcing in improving the fluency and accuracy of generated responses in multimodal dialogue systems?
• RQ4 Can data augmentation via shuffling dialog history QA pairs lead to measurable performance gains in multimodal QA?
• RQ5 Which combination of pooling strategies (max vs. average) and RNN variants (BiGRU vs. BiLSTM) yields the best generation quality?

Key findings

• Teacher forcing led to a 30% to 300% improvement in all evaluation metrics (BLEU, CIDEr, etc.) compared to the baseline model without it.
• Max pooling outperformed average pooling, significantly increasing BLEU and CIDEr scores by emphasizing dominant features across the dialog history.
• The proposed data augmentation technique, which increases the dataset size by a factor of 2 through shuffling QA pairs, further improved model performance.
• BiGRUs achieved performance comparable to BiLSTMs, indicating no significant advantage of LSTMs over GRUs in this specific setup.
• Scheduled sampling did not improve performance over teacher forcing and introduced training instability due to noisy predictions.
• The model achieved a CIDEr score of 0.224 without teacher forcing, which increased substantially after applying teacher forcing and data augmentation.