[Paper Review] Uncovering Temporal Context for Video Question and Answering

Linchao Zhu, Zhongwen Xu+2 more|arXiv (Cornell University)|Nov 15, 2015

Multimodal Machine Learning Applications52 references41 citations

TL;DR

This paper proposes a GRU-based encoder-decoder framework with dual-channel ranking loss for video question answering across past, present, and future temporal states. By leveraging joint visual-linguistic representations and a large-scale dataset of 109,895 video clips with 390,744 multiple-choice questions, the method significantly outperforms baselines, achieving 78.3% and 79.7% accuracy on TACoS for past inference and future prediction under hard examples, respectively.

ABSTRACT

In this work, we introduce Video Question Answering in temporal domain to infer the past, describe the present and predict the future. We present an encoder-decoder approach using Recurrent Neural Networks to learn temporal structures of videos and introduce a dual-channel ranking loss to answer multiple-choice questions. We explore approaches for finer understanding of video content using question form of "fill-in-the-blank", and managed to collect 109,895 video clips with duration over 1,000 hours from TACoS, MPII-MD, MEDTest 14 datasets, while the corresponding 390,744 questions are generated from annotations. Extensive experiments demonstrate that our approach significantly outperforms the compared baselines.

Motivation & Objective

• To address the lack of temporal reasoning in video question answering by enabling inference about past actions, present states, and future predictions.
• To improve video understanding beyond video captioning by modeling fine-grained interactions between video frames and natural language questions.
• To develop a scalable, end-to-end framework that jointly learns visual and linguistic representations for temporal video QA.
• To create and release a large-scale, diverse video QA dataset with 1,000+ hours of video and 390K multiple-choice questions for benchmarking.
• To evaluate the model using a 'fill-in-the-blank' (FITB) format with controlled difficulty to enable reliable quantitative comparison.

Proposed method

• Uses a GRU-based encoder-decoder architecture to model long-range temporal dependencies in video clips.
• Employs a dual-channel ranking loss to jointly optimize for past inference, present description, and future prediction tasks.
• Integrates visual features from ConvNets with word and sentence embeddings in a joint embedding space to enhance cross-modal understanding.
• Leverages external knowledge bases (e.g., BookCorpus, Google News) to improve question parsing and reasoning.
• Trains the model in an unsupervised manner on video clips to learn temporal structures before fine-tuning on QA tasks.
• Uses a 'fill-in-the-blank' multiple-choice format for evaluation, enabling controlled and reproducible assessment of model performance.

Experimental results

Research questions

• RQ1 Can a unified video QA framework effectively model temporal reasoning across past, present, and future states?
• RQ2 How does joint visual-linguistic representation learning improve video QA performance compared to isolated modality modeling?
• RQ3 To what extent does the dual-channel ranking loss enhance answer selection accuracy across different temporal reasoning tasks?
• RQ4 Does the GRU-based encoder-decoder architecture outperform ConvNet-based models in modeling long-range temporal dependencies in video?
• RQ5 Can a large-scale, multiple-choice video QA dataset with controlled difficulty enable reliable and scalable evaluation of temporal video understanding models?

Key findings

• The proposed GRU-based model achieves 78.3% accuracy on TACoS for past inference and 79.7% for future prediction under hard examples, outperforming ConvNet baselines.
• The model improves over ConvNet baselines by 3.5% on past inference and 2.8% on future prediction in the TACoS dataset under hard examples.
• On MPII-MD, the model achieves 72.1% accuracy for past inference and 73.6% for future prediction under hard examples, showing consistent gains over ConvNets.
• The model performs better on future prediction than past inference, likely due to shorter-term dependencies in future prediction tasks.
• The dual-channel ranking loss effectively improves answer selection by leveraging both visual and linguistic context across all three temporal tasks.
• The model demonstrates robustness to overfitting due to reduced parameter count in GRUs and effective joint learning of visual and linguistic features.