[Paper Review] Video Description: A Survey of Methods, Datasets and Evaluation Metrics

Nayyer Aafaq, Ajmal Mian+3 more|UWA Profiles and Research Repository (UWA)|Jun 1, 2018

Multimodal Machine Learning Applications38 references95 citations

TL;DR

A comprehensive survey of video description research, tracing classical, statistical, and deep learning methods; compares datasets and evaluation metrics; discusses challenges and future directions.

ABSTRACT

Video description is the automatic generation of natural language sentences that describe the contents of a given video. It has applications in human-robot interaction, helping the visually impaired and video subtitling. The past few years have seen a surge of research in this area due to the unprecedented success of deep learning in computer vision and natural language processing. Numerous methods, datasets and evaluation metrics have been proposed in the literature, calling the need for a comprehensive survey to focus research efforts in this flourishing new direction. This paper fills the gap by surveying the state of the art approaches with a focus on deep learning models; comparing benchmark datasets in terms of their domains, number of classes, and repository size; and identifying the pros and cons of various evaluation metrics like SPICE, CIDEr, ROUGE, BLEU, METEOR, and WMD. Classical video description approaches combined subject, object and verb detection with template based language models to generate sentences. However, the release of large datasets revealed that these methods can not cope with the diversity in unconstrained open domain videos. Classical approaches were followed by a very short era of statistical methods which were soon replaced with deep learning, the current state of the art in video description. Our survey shows that despite the fast-paced developments, video description research is still in its infancy due to the following reasons. Analysis of video description models is challenging because it is difficult to ascertain the contributions, towards accuracy or errors, of the visual features and the adopted language model in the final description. Existing datasets neither contain adequate visual diversity nor complexity of linguistic structures. Finally, current evaluation metrics ...

Motivation & Objective

• Survey the evolution of video description methods from classical to deep learning.
• Compare benchmark datasets in terms of domain, size, and diversity.
• Analyze evaluation metrics and their correlation with human judgments.
• Identify current limitations in datasets and metrics and propose future research directions.

Proposed method

• Classify video description methods into classical SVO/template-based, statistical, and deep learning approaches.
• Describe architectural trends such as CNN-LSTM/GRU encoders, attention, and semantic attributes.
• Discuss dataset characteristics and how large open-domain datasets drive method development.
• Review evaluation metrics (BLEU, ROUGE, METEOR, CIDEr, SPICE, WMD) and their alignment with human judgments.

Experimental results

Research questions

• RQ1 What are the main methodological phases in video description evolution and their limitations?
• RQ2 How do benchmark datasets differ in content, complexity, and scale for video description?
• RQ3 What are the strengths and weaknesses of current evaluation metrics for video descriptions?
• RQ4 What future directions can address dataset diversity and metric alignment with human judgments.

Key findings

• Video description has evolved from template-based to deep learning methods aided by large multi-modal datasets.
• Open-domain and longer videos expose vocabulary and linguistic complexity that early methods could not handle.
• Evaluation metrics differ in what they measure and often do not perfectly align with human judgments.
• Current metrics like BLEU, METEOR, ROUGE, CIDEr, SPICE, and WMD cover different aspects of description quality and have instability issues.