[Paper Review] Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog

Shachi H Kumar, Eda Okur+4 more|arXiv (Cornell University)|Dec 20, 2018

Multimodal Machine Learning Applications18 references9 citations

TL;DR

This paper proposes an end-to-end audio-visual scene-aware dialog system that enhances context understanding through topic modeling, multi-head attention over dialog history, and end-to-end audio classification via AclNet. It demonstrates improved performance over the baseline on multiple metrics, particularly in audio-related question answering, with AclNet features outperforming VGGish on both general and audio-specific evaluations.

ABSTRACT

With the recent advancements in AI, Intelligent Virtual Assistants (IVA) have become a ubiquitous part of every home. Going forward, we are witnessing a confluence of vision, speech and dialog system technologies that are enabling the IVAs to learn audio-visual groundings of utterances and have conversations with users about the objects, activities and events surrounding them. As a part of the 7th Dialog System Technology Challenges (DSTC7), for Audio Visual Scene-Aware Dialog (AVSD) track, We explore `topics' of the dialog as an important contextual feature into the architecture along with explorations around multimodal Attention. We also incorporate an end-to-end audio classification ConvNet, AclNet, into our models. We present detailed analysis of the experiments and show that some of our model variations outperform the baseline system presented for this task.

Motivation & Objective

• To improve context awareness in audio-visual scene-aware dialog systems by incorporating dialog topic modeling.
• To enhance response generation by applying multi-head attention over dialog history and multimodal features.
• To explore end-to-end audio feature extraction using AclNet for better audio understanding in dialog systems.
• To evaluate the impact of contextual and multimodal features on response quality and binary answer prediction.
• To compare AclNet and VGGish audio features in terms of performance on audio-related dialog turns.

Proposed method

• Topic modeling is applied to dialog history and questions using seed-words to identify latent conversation topics.
• Multi-head attention is applied to the hidden states of dialog history LSTMs to allow selective focus on relevant past utterances.
• AclNet, a 50-class end-to-end audio classification ConvNet, is used to extract audio embeddings from raw audio features.
• Attention mechanisms are extended to jointly attend over dialog history states and multimodal audio/video features.
• The model integrates topic representations, dialog history states, and audio features into a unified encoder-decoder framework with cross-attention.
• Binary answer evaluation is used to assess model performance on yes/no questions, particularly for audio-related queries.

Experimental results

Research questions

• RQ1 Can topic modeling based on seed-words improve context understanding in audio-visual dialog systems?
• RQ2 Does attention over multiple hidden states of dialog history improve response generation compared to relying only on the final LSTM state?
• RQ3 Can end-to-end audio classification via AclNet outperform traditional audio embeddings like VGGish in scene-aware dialog tasks?
• RQ4 How do different attention mechanisms (on history vs. history + audio features) affect response quality and answer accuracy?
• RQ5 Does incorporating audio-specific features lead to better performance on audio-related questions compared to baseline models?

Key findings

• The model with attention over all LSTM hidden states of dialog history outperformed the baseline on BLEU3, BLEU4, METEOR, CIDEr, and ROUGE scores.
• The configuration using attention over history states and multimodal features showed improved CIDEr scores but slightly degraded performance on binary answer evaluation compared to the history-only attention variant.
• AclNet features improved all evaluation metrics over the baseline, and outperformed VGGish on audio-related question subsets.
• Qualitative analysis showed that B+AclNet correctly answered 'is there sound in the video?' while B+VGGish failed, and better captured subtle audio events like 'sneeze'.
• Topic modeling with seed-words improved overall performance compared to the baseline, indicating that guided topic modeling enhances contextual grounding.
• The model with AclNet and attention over history states achieved the best overall performance, especially on audio-specific queries.