[Paper Review] Learning Language-Visual Embedding for Movie Understanding with Natural-Language

Atousa Torabi, Niket Tandon+1 more|arXiv (Cornell University)|Sep 26, 2016

Multimodal Machine Learning ApplicationsComputer Science26 references72 citations

TL;DR

This paper proposes a joint language-visual embedding model for movie understanding using natural language, leveraging multimodal deep learning to improve video annotation and retrieval. By training on the LSMDC16 dataset with rephrased captions and human-activity predicates, the best model achieves 19.2% Recall@10 on video annotation and 58.11% accuracy on a novel multiple-choice test, demonstrating strong performance in visual-language understanding with structured evaluation.

ABSTRACT

Learning a joint language-visual embedding has a number of very appealing properties and can result in variety of practical application, including natural language image/video annotation and search. In this work, we study three different joint language-visual neural network model architectures. We evaluate our models on large scale LSMDC16 movie dataset for two tasks: 1) Standard Ranking for video annotation and retrieval 2) Our proposed movie multiple-choice test. This test facilitate automatic evaluation of visual-language models for natural language video annotation based on human activities. In addition to original Audio Description (AD) captions, provided as part of LSMDC16, we collected and will make available a) manually generated re-phrasings of those captions obtained using Amazon MTurk b) automatically generated human activity elements in "Predicate + Object" (PO) phrases based on "Knowlywood", an activity knowledge mining model. Our best model archives Recall@10 of 19.2% on annotation and 18.9% on video retrieval tasks for subset of 1000 samples. For multiple-choice test, our best model achieve accuracy 58.11% over whole LSMDC16 public test-set.

Motivation & Objective

• To develop a joint language-visual embedding model that enables accurate video annotation and retrieval using natural language queries.
• To address the challenge of evaluating visual-language models in video understanding with a new, automated, and precise multiple-choice test based on human activities.
• To improve model generalization by training on a combination of COCO and LSMDC16 datasets, including rephrased captions and automatically extracted predicate-object phrases.
• To evaluate model performance on both standard ranking tasks and the proposed multiple-choice test for more objective and scalable assessment.

Proposed method

• The authors design three joint language-visual neural network architectures: one using soft attention over global video features (M1), and two using LSTM-based encoding with soft attention (M2 and M3).
• A temporal attention network is employed to dynamically highlight salient video frames based on phrase queries, improving retrieval precision.
• The model is trained using a combination of COCO image captions and LSMDC16 video descriptions, including manually rephrased captions collected via Amazon Mechanical Turk.
• Human activity elements are extracted using the Knowlywood model to generate "Predicate + Object" (PO) phrases for improved grounding of actions in video.
• For evaluation, the authors introduce a multiple-choice test based on video clips and natural language questions, with answers derived from PO phrases and rephrased captions.
• The model uses order-embedding loss and pairwise ranking loss to optimize for both retrieval and classification performance.

Experimental results

Research questions

• RQ1 Can a joint language-visual embedding model effectively improve video annotation and retrieval performance on the LSMDC16 dataset?
• RQ2 How does training on rephrased captions and automatically extracted predicate-object phrases affect model generalization and performance?
• RQ3 Can a multiple-choice test based on human activities provide a reliable, automated evaluation benchmark for visual-language models in video understanding?
• RQ4 What is the impact of combining COCO and LSMDC16 datasets on model performance in zero-shot and few-shot settings?

Key findings

• The best-performing model, M2 trained on COCO and LSMDC16 with rephrased captions (C+L’16+RP), achieves 19.2% Recall@10 on video annotation and 18.9% Recall@10 on video retrieval for a 1000-sample subset.
• On the full LSMDC16 public test set, the model achieves 58.11% accuracy on the proposed multiple-choice test, outperforming other variants.
• Training with the annotation ranking (AR) loss improves multiple-choice accuracy by 2% compared to pairwise ranking loss.
• The model demonstrates strong performance in phrase-based video search, with attention networks correctly highlighting salient frames corresponding to query phrases.
• The combination of COCO and LSMDC16 data leads to better generalization, as shown by improved median rank (medR) and higher recall across all settings.