[Paper Review] MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description

Oliver Nina, Washington García+2 more|arXiv (Cornell University)|Sep 19, 2018

Multimodal Machine Learning Applications34 references3 citations

TL;DR

This paper proposes MTLE, a multitask learning encoder-decoder framework that improves video and movie description by jointly training a bi-directional RNN encoder with multiple decoders using a semantic distance function to select diverse caption pairs. The method achieves state-of-the-art performance on multiple benchmarks and was ranked highest by human evaluators for usefulness to the visually impaired in the LSMDC 2017 challenge.

ABSTRACT

Learning visual feature representations for video analysis is a daunting task that requires a large amount of training samples and a proper generalization framework. Many of the current state of the art methods for video captioning and movie description rely on simple encoding mechanisms through recurrent neural networks to encode temporal visual information extracted from video data. In this paper, we introduce a novel multitask encoder-decoder framework for automatic semantic description and captioning of video sequences. In contrast to current approaches, our method relies on distinct decoders that train a visual encoder in a multitask fashion. Our system does not depend solely on multiple labels and allows for a lack of training data working even with datasets where only one single annotation is viable per video. Our method shows improved performance over current state of the art methods in several metrics on multi-caption and single-caption datasets. To the best of our knowledge, our method is the first method to use a multitask approach for encoding video features. Our method demonstrates its robustness on the Large Scale Movie Description Challenge (LSMDC) 2017 where our method won the movie description task and its results were ranked among other competitors as the most helpful for the visually impaired.

Motivation & Objective

• Address the challenge of learning robust visual feature representations for video captioning with limited or sparse training annotations.
• Overcome the bias and poor generalization of single-decoder RNN-based encoders in video captioning by introducing a multitask learning framework.
• Enable effective training and improved performance on datasets with only one caption per video, such as LSMDC, by leveraging semantic diversity through a distance-based loss.
• Improve model generalization by jointly training the encoder with multiple decoders that project captions into a shared semantic space.
• Develop a framework that performs well under both automatic metrics and human evaluation, particularly for real-world accessibility applications.

Proposed method

• Use a CNN to extract visual features from each video frame, followed by a bi-directional RNN-based encoder to produce a single visual feature encoding.
• Train the encoder jointly with multiple decoders using a multitask learning setup, where each decoder is conditioned on the encoder output and predicts a caption.
• Apply a semantic distance function (SDF) to select caption pairs with the maximum semantic distance for each video during training, ensuring diversity in supervision.
• Formulate a novel multitask loss function that combines supervision from multiple caption pairs, enabling regularization and improved generalization even with single-caption data.
• Use soft attention mechanisms in the decoders to dynamically attend to relevant visual features during caption generation.
• Optimize the encoder and decoders end-to-end via back-propagation, with shared encoder weights across all decoders to maintain consistency in visual representation learning.

Experimental results

Research questions

• RQ1 Can a multitask learning framework improve the generalization of visual feature encoders in video captioning when training data is sparse or limited to one caption per video?
• RQ2 Does selecting caption pairs with maximum semantic distance during training lead to better visual representation learning and improved caption quality?
• RQ3 How does the proposed MTLE framework compare to state-of-the-art methods in terms of automatic metrics and human evaluation, especially for accessibility applications?
• RQ4 Can a single encoder be effectively shared across multiple decoders trained on diverse caption pairs without performance degradation?
• RQ5 To what extent does the multitask loss function reduce bias toward individual captions and improve robustness in zero-shot or low-resource settings?

Key findings

• MTLE achieved the highest human evaluation score (2.50/5.0) among all competing methods in the LSMDC 2017 challenge, outperforming top methods like Fcrerank (2.18) and PostProp (2.17) in perceived usefulness for the visually impaired.
• On the LSMDC 2017 dataset, MTLE achieved a CIDEr score of 0.073, outperforming several strong competitors including FuseNet (0.083) and Attn2l (0.073), with a BLEU score of 0.003 and METEOR of 0.052.
• The method demonstrated superior robustness, with the smallest percentage of both minor and major errors in human judgment histograms, indicating higher reliability in caption quality.
• Qualitative results showed that MTLE-generated captions often matched or approached human-level performance, with some cases correctly identifying attributes like gender not present in the ground truth.
• MTLE showed strong generalization on multiple datasets, including MSVD, MSR-VTT, TRECVID, and LSMDC, with consistent improvements over baselines across multiple metrics.
• The framework successfully handled single-caption datasets by using a regularization term in the loss function that leverages semantic diversity even when only one caption is available per video.