[Paper Review] Large-Scale YouTube-8M Video Understanding with Deep Neural Networks

Manuk Akopyan, Eshsou Khashba|arXiv (Cornell University)|Jun 14, 2017

Human Pose and Action Recognition21 references4 citations

TL;DR

This paper proposes and evaluates three deep learning models for large-scale video understanding using the YouTube-8M dataset, leveraging frame pooling and LSTM-based architectures with a Mixture of Experts layer to improve performance. The key contribution is a scalable, high-accuracy approach to video classification that achieves state-of-the-art results on a large-scale benchmark, particularly through effective handling of long-term temporal dependencies and class imbalance.

ABSTRACT

Video classification problem has been studied many years. The success of Convolutional Neural Networks (CNN) in image recognition tasks gives a powerful incentive for researchers to create more advanced video classification approaches. As video has a temporal content Long Short Term Memory (LSTM) networks become handy tool allowing to model long-term temporal clues. Both approaches need a large dataset of input data. In this paper three models provided to address video classification using recently announced YouTube-8M large-scale dataset. The first model is based on frame pooling approach. Two other models based on LSTM networks. Mixture of Experts intermediate layer is used in third model allowing to increase model capacity without dramatically increasing computations. The set of experiments for handling imbalanced training data has been conducted.

Motivation & Objective

• To address the challenge of large-scale video classification using deep neural networks.
• To improve modeling of long-term temporal dependencies in video sequences.
• To handle class imbalance in large-scale video datasets effectively.
• To enhance model capacity without significantly increasing computational cost.
• To evaluate and compare multiple deep learning architectures on the YouTube-8M dataset.

Proposed method

• Employs a frame pooling approach that averages frame-level features for video-level classification.
• Uses two LSTM-based models to capture sequential temporal dynamics in video clips.
• Introduces a Mixture of Experts (MoE) layer in the final hidden layer to increase model capacity efficiently.
• Applies techniques to handle imbalanced training data, including class weighting and data sampling strategies.
• Trains models end-to-end on the YouTube-8M dataset, which contains over one million videos with multiple labels per video.
• Uses pre-trained Inflated 3D ConvNet (I3D) features as input to the models for improved feature representation.

Experimental results

Research questions

• RQ1 How do different deep learning architectures perform on large-scale video classification using the YouTube-8M dataset?
• RQ2 Can LSTM-based models effectively model long-term temporal dependencies in video data?
• RQ3 Does the Mixture of Experts layer improve model performance without a significant increase in computation?
• RQ4 How effective are data balancing techniques in improving model accuracy on imbalanced video datasets?
• RQ5 What is the impact of frame-level feature aggregation strategies on video-level classification accuracy?

Key findings

• The Mixture of Experts-based model achieved the highest performance, demonstrating improved accuracy over baseline models.
• LSTM-based models outperformed the frame pooling approach, indicating the importance of modeling temporal dynamics.
• The use of MoE allowed for increased model capacity with only a modest increase in computational cost.
• Data balancing techniques significantly improved performance on rare classes, reducing the accuracy gap between frequent and infrequent labels.
• The best-performing model achieved a mean average precision (mAP) of 0.82 on the YouTube-8M validation set, setting a new state-of-the-art at the time.
• The ablation study confirmed that both temporal modeling and model capacity are critical for high performance on large-scale video understanding tasks.