[Paper Review] Sequence to Sequence -- Video to Text

Subhashini Venugopalan, Marcus Rohrbach+4 more|arXiv (Cornell University)|May 3, 2015

Multimodal Machine Learning Applications43 references193 citations

TL;DR

This paper proposes S2VT, a sequence-to-sequence deep learning model that maps variable-length video inputs (frames) to natural language captions using stacked LSTMs. It achieves state-of-the-art performance on MSVD, MPII-MD, and M-VAD datasets by jointly learning temporal video dynamics and language modeling without explicit attention, outperforming prior methods on METEOR scores by up to 1.5 percentage points.

ABSTRACT

Real-world videos often have complex dynamics; and methods for generating open-domain video descriptions should be sensitive to temporal structure and allow both input (sequence of frames) and output (sequence of words) of variable length. To approach this problem, we propose a novel end-to-end sequence-to-sequence model to generate captions for videos. For this we exploit recurrent neural networks, specifically LSTMs, which have demonstrated state-of-the-art performance in image caption generation. Our LSTM model is trained on video-sentence pairs and learns to associate a sequence of video frames to a sequence of words in order to generate a description of the event in the video clip. Our model naturally is able to learn the temporal structure of the sequence of frames as well as the sequence model of the generated sentences, i.e. a language model. We evaluate several variants of our model that exploit different visual features on a standard set of YouTube videos and two movie description datasets (M-VAD and MPII-MD).

Motivation & Objective

• To address the challenge of generating open-domain, natural language descriptions for videos with variable-length input and output sequences.
• To model both the temporal structure of video frames and the sequential grammar of natural language descriptions in an end-to-end trainable framework.
• To improve video captioning performance by jointly learning visual representation and language generation without relying on template-based or two-stage pipelines.
• To evaluate the model on diverse datasets including YouTube videos and movie corpora, demonstrating robustness and generalization.
• To show that a simple sequence-to-sequence architecture with LSTMs and multimodal visual inputs (RGB and optical flow) can outperform complex attention-based models.

Proposed method

• The model uses a stacked Long Short-Term Memory (LSTM) network to encode a sequence of video frames into a hidden state representation.
• Each frame is processed through a pre-trained Convolutional Neural Network (CNN), such as VGG, to extract visual features, which are then fed sequentially into the encoder LSTM.
• Optical flow features between consecutive frames are also extracted and processed by a separate CNN before being fed into the encoder LSTM to capture motion dynamics.
• The final hidden state of the encoder is used to initialize the decoder LSTM, which generates a sequence of words autoregressively, one word at a time.
• The entire model is trained end-to-end using cross-entropy loss to minimize the difference between predicted and ground-truth caption sequences.
• Dropout is applied at the input and output layers of both encoder and decoder LSTMs to prevent overfitting, especially on smaller movie datasets.

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence model with LSTMs effectively learn temporal dependencies in video frames and generate grammatically correct, descriptive sentences?
• RQ2 Does the joint learning of visual encoding and language generation outperform two-stage or template-based captioning approaches?
• RQ3 How does the inclusion of optical flow features improve video captioning performance compared to RGB-only inputs?
• RQ4 Can the model generalize across diverse video domains, such as YouTube clips and movie scenes, without architectural modifications?
• RQ5 To what extent does the model’s performance scale with additional training data, and how does it compare to state-of-the-art methods on benchmark datasets?

Key findings

• On the MSVD dataset, S2VT achieves a METEOR score of 18.5%, outperforming previous methods and demonstrating strong performance on a standard YouTube video captioning benchmark.
• On the MPII-MD dataset, S2VT achieves a METEOR score of 7.1%, surpassing the SMT baseline (5.6%) and mean pooling baselines (6.7%), showing significant improvement on a challenging movie description dataset.
• On the M-VAD dataset, S2VT achieves a METEOR score of 6.7%, outperforming the temporal attention model (4.3%) and mean pooling (6.1%), indicating superior generalization on complex movie scenes.
• On the combined LSMDC challenge, S2VT achieves a METEOR score of 7.0% on the public test set, demonstrating strong performance across diverse movie descriptions.
• 42.9% of generated sentences on MSVD exactly match a training sentence, and 81.2% are within one edit distance, indicating high relevance and linguistic quality.
• The model shows high model capacity and generalization, with performance improving significantly on larger datasets, suggesting scalability and robustness to domain shifts.