[Paper Review] Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Subhashini Venugopalan, Huijuan Xu+4 more|arXiv (Cornell University)|Dec 15, 2014

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes an end-to-end deep neural network that translates video clips directly into natural language sentences using a unified architecture combining convolutional neural networks (CNNs) for visual feature extraction and long short-term memory (LSTM) networks for sequence generation. By transferring knowledge from large-scale image classification and image captioning datasets, the model achieves state-of-the-art performance on video description tasks, significantly improving fluency, accuracy, and relevance compared to prior methods.

ABSTRACT

Solving the visual symbol grounding problem has long been a goal of artificial intelligence. The field appears to be advancing closer to this goal with recent breakthroughs in deep learning for natural language grounding in static images. In this paper, we propose to translate videos directly to sentences using a unified deep neural network with both convolutional and recurrent structure. Described video datasets are scarce, and most existing methods have been applied to toy domains with a small vocabulary of possible words. By transferring knowledge from 1.2M+ images with category labels and 100,000+ images with captions, our method is able to create sentence descriptions of open-domain videos with large vocabularies. We compare our approach with recent work using language generation metrics, subject, verb, and object prediction accuracy, and a human evaluation.

Motivation & Objective

• To address the challenge of generating natural language descriptions for open-domain, in-the-wild videos with large vocabularies and limited annotated training data.
• To overcome limitations of prior methods that rely on intermediate semantic role representations or rigid sentence templates.
• To improve video description quality by transferring knowledge from large-scale image and caption datasets to the video domain.
• To develop an end-to-end trainable model that jointly learns visual meaning states and fluent language generation without intermediate supervision.
• To evaluate the model’s performance using automated metrics, zero-shot transfer, and human evaluation on the YouTube video corpus.

Proposed method

• The model uses a pre-trained convolutional neural network (CIFAR-10/ILSVRC2012) to extract spatial features from each video frame.
• Visual features from all frames are temporally pooled (mean pooling) to form a compact video-level representation.
• A deep Long Short-Term Memory (LSTM) recurrent network generates natural language sentences token-by-token, conditioned on the pooled visual features.
• The LSTM is pre-trained on image captioning datasets (Flickr30k and COCO) to learn rich linguistic patterns and semantic composition.
• Knowledge transfer is achieved by initializing the video-to-text model with weights from image captioning and image classification pre-training.
• The entire network is fine-tuned end-to-end on the YouTube video description dataset to adapt to video-specific features and language patterns.

Experimental results

Research questions

• RQ1 Can an end-to-end deep neural network effectively translate raw video pixels into fluent, grammatically correct natural language sentences?
• RQ2 How does knowledge transfer from image classification and image captioning datasets improve performance on video description with limited video-specific training data?
• RQ3 To what extent does mean pooling of frame-level features improve video description quality compared to using individual frames?
• RQ4 Does pre-training on large-scale image and caption data lead to better subject, verb, and object prediction accuracy in video descriptions?
• RQ5 How does the proposed model compare to template-based or role-based approaches in terms of fluency, grammatical correctness, and human relevance?

Key findings

• The proposed end-to-end CNN-LSTM model outperforms previous state-of-the-art methods on the YouTube video description dataset in both automated metrics and human evaluation.
• Pre-training on COCO and Flickr30k image captioning datasets significantly improves subject, verb, and object prediction accuracy compared to training only on video data.
• The model pre-trained on COCO2014 shows the largest performance gain, indicating that large-scale image captioning data is highly effective for video description.
• Mean pooling of frame features leads to significantly better performance than using individual frames, suggesting it captures more representative video-level semantics.
• Human evaluation shows that generated sentences are more relevant to video content than previous models, though grammatical correctness remains a challenge compared to template-based systems.
• Transfer learning from image captioning data reduces the performance gap between model outputs and human-annotated descriptions, demonstrating the value of auxiliary data.