[Paper Review] YouTube-8M: A Large-Scale Video Classification Benchmark

Sami Abu-El-Haija, Nisarg Kothari+5 more|arXiv (Cornell University)|Sep 27, 2016

Multimodal Machine Learning Applications32 references920 citations

TL;DR

This paper introduces YouTube-8M, a large-scale multi-label video classification benchmark with ~8.3M videos (500k+ hours) and 4,800 labels, plus pre-extracted frame features and baselines. It evaluates frame-based and video-level representations and shows transfer to Sports-1M and ActivityNet.

ABSTRACT

Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.

Motivation & Objective

• Introduce a large-scale, general multi-label video classification benchmark grounded in YouTube data.
• Provide a visually recognizable vocabulary of 4,800 Knowledge Graph entities spanning diverse top-level categories.
• Offer pre-computed frame-level features and standardized train/validation/test splits to enable scalable research.
• Demonstrate baseline models on fixed frame features and fixed video representations, and explore transfer learning to other benchmarks.

Proposed method

• Construct a visual, multi-label vocabulary of ~10,000 visually recognizable entities (filtered to ≥200 videos).
• Collect ~8.26 million videos (≈500k hours) with 1,400+ frames per video for feature extraction and annotation.
• Decode videos at 1 frame per second; extract 2048-dim pool_3/_reshape features from Inception; apply PCA+whitening to 1024 dims; 8x compression via 8-bit quantization.
• Provide fixed frame-level features for all videos and label partitions; releasetrain/validate/test splits (train:validate:test = 5,786,881:1,652,167:825,602).
• Train simple frame-based and video-level models: one-vs-all logistic classifiers, online SVM with hinge loss, and mixture-of-experts variants; explore Deep Bag-of-Frames (DBoF) and LSTM on frame features.
• Explore video-level representations by aggregating frame features (mean, std, top-K ordinal statistics) and normalizing with PCA whitening; train binary classifiers on these compact representations.

Experimental results

Research questions

• RQ1 Can a large-scale, diverse, multi-label video dataset enable learning general video representations beyond action-centric benchmarks?
• RQ2 How well do fixed frame-level features and fixed video-level representations support scalable multi-label video classification at this scale?
• RQ3 Do representations learned on YouTube-8M transfer to other benchmarks like Sports-1M and ActivityNet?
• RQ4 What is the impact of model choices (logistic regression, hinge-loss SVM, mixture of experts, LSTM) on multi-label video classification performance?
• RQ5 How does dataset scale and label noise affect evaluation and baselines?

Key findings

• YouTube-8M contains ~8.26 million videos, 4,800 classes, and ≈1.9 billion frames after processing the first 6 minutes at 1 FPS.
• Pre-computed frame features (2048-dims) with PCA+ whitening and 8-bit quantization enable scalable baselines without heavy computation for researchers.
• Baseline models on fixed frame features and video-level representations are trainable on a single machine with TensorFlow and converge in under a day on this data.
• Video representations learned on YouTube-8M generalize to other benchmarks such as Sports-1M and ActivityNet, with notable improvements on ActivityNet (mAP from 53.8% to 77.6%).
• Human-rated test subset indicates precision 78.8% and recall 14.5% for ground-truth labels, highlighting missing-label challenges and opportunities for modeling incorrect or missing labels.