[Paper Review] Gaining Extra Supervision via Multi-task learning for Multi-Modal Video Question Answering

Jun-Yeong Kim, Minuk Ma+3 more|arXiv (Cornell University)|May 28, 2019

Multimodal Machine Learning Applications40 references4 citations

TL;DR

This paper proposes a multi-task learning framework for multi-modal video question answering that gains extra supervision by jointly training a video question answering network with two auxiliary tasks: temporal localization and modality alignment. By hierarchically sharing features and using multi-task ratio scheduling inspired by curriculum learning, the method achieves state-of-the-art performance of 67.05% on the TVQA benchmark, demonstrating significant gains through synergistic learning signals.

ABSTRACT

This paper proposes a method to gain extra supervision via multi-task learning for multi-modal video question answering. Multi-modal video question answering is an important task that aims at the joint understanding of vision and language. However, establishing large scale dataset for multi-modal video question answering is expensive and the existing benchmarks are relatively small to provide sufficient supervision. To overcome this challenge, this paper proposes a multi-task learning method which is composed of three main components: (1) multi-modal video question answering network that answers the question based on the both video and subtitle feature, (2) temporal retrieval network that predicts the time in the video clip where the question was generated from and (3) modality alignment network that solves metric learning problem to find correct association of video and subtitle modalities. By simultaneously solving related auxiliary tasks with hierarchically shared intermediate layers, the extra synergistic supervisions are provided. Motivated by curriculum learning, multi task ratio scheduling is proposed to learn easier task earlier to set inductive bias at the beginning of the training. The experiments on publicly available dataset TVQA shows state-of-the-art results, and ablation studies are conducted to prove the statistical validity.

Motivation & Objective

• To address the limited supervision in existing multi-modal video question answering benchmarks, which are too small for effective deep learning.
• To improve video question answering performance by leveraging auxiliary tasks that provide additional supervisory signals.
• To design a multi-task learning framework that hierarchically shares features between the main QA task and two auxiliary tasks: modality alignment and temporal localization.
• To introduce multi-task ratio scheduling that prioritizes easier tasks early in training to establish inductive bias, inspired by curriculum learning.
• To empirically validate the effectiveness of the proposed method through ablation studies and state-of-the-art results on the TVQA dataset.

Proposed method

• The method employs a multi-modal video question answering network that fuses video and subtitle features to answer questions.
• It introduces a temporal retrieval network that predicts the relevant moment in the video clip corresponding to the question, sharing higher-level features with the QA network.
• A modality alignment network is used to solve a metric learning problem, ensuring correct pairing of video and subtitle features, and shares lower-level features with the QA network.
• The model uses hierarchical parameter sharing: modality alignment shares early layers, while temporal localization shares deeper layers with the QA network.
• Multi-task ratio scheduling is applied to control the training objective ratio, prioritizing easier tasks (modality alignment) at the start to guide learning.
• The framework is trained end-to-end with a combined loss function that balances the main QA task and the two auxiliary tasks.

Experimental results

Research questions

• RQ1 Can auxiliary tasks such as temporal localization and modality alignment improve performance in multi-modal video question answering?
• RQ2 Does multi-task learning with hierarchical feature sharing provide synergistic supervision that boosts the main QA task?
• RQ3 Can curriculum-inspired multi-task ratio scheduling improve training stability and performance by prioritizing easier tasks early?
• RQ4 How does the performance of the proposed method compare to existing state-of-the-art models on the TVQA benchmark?
• RQ5 What is the relative contribution of each auxiliary task (modality alignment vs. temporal localization) to the overall performance gain?

Key findings

• The proposed method achieves a state-of-the-art test accuracy of 67.05% on the TVQA dataset using visual concept features, outperforming the previous best model by 1.59 percentage points.
• The subtitle-only model (S+Q) achieves 64.63% accuracy, surpassing the TVQA S+V+Q baseline with ImageNet features (63.57%), demonstrating the effectiveness of extra supervision.
• The ablation study shows that both temporal localization and modality alignment contribute positively, with modality alignment providing a larger performance gain than temporal localization.
• The full model with visual concept features and both auxiliary tasks achieves 66.22% validation accuracy, which is 0.85 percentage points higher than the baseline QA model.
• Multi-task ratio scheduling improves training dynamics by setting inductive bias early, as evidenced by consistent performance gains across ablation settings.
• The results confirm that multi-task learning with carefully designed auxiliary tasks can significantly enhance performance even in low-supervision settings.