[Paper Review] Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification

Po-Yao Huang, Ye Yuan+3 more|arXiv (Cornell University)|Jul 5, 2017

Multimodal Machine Learning Applications22 references3 citations

TL;DR

This paper presents a hybrid deep learning framework for large-scale multi-label video classification on the YouTube-8M dataset, combining refined video representation learning via residual and hypercolumn networks, latent concept mining to model label dependencies, and a weighted ensemble of video-level and frame-level models with temporal segment augmentation. The approach achieves 84.68% GAP on the validation set and 84.66% on the official test set, outperforming prior methods by effectively capturing complex label relationships and leveraging complementary model types.

ABSTRACT

We report on CMU Informedia Lab's system used in Google's YouTube 8 Million Video Understanding Challenge. In this multi-label video classification task, our pipeline achieved 84.675% and 84.662% GAP on our evaluation split and the official test set. We attribute the good performance to three components: 1) Refined video representation learning with residual links and hypercolumns 2) Latent concept mining which captures interactions among concepts. 3) Learning with temporal segments and weighted multi-model ensemble. We conduct experiments to validate and analyze the contribution of our models. We also share some unsuccessful trials leveraging conventional approaches such as recurrent neural networks for video representation learning for this large-scale video dataset. All the codes to reproduce our results are publicly available at https://github.com/Martini09/informedia-yt8m-release.

Motivation & Objective

• To improve large-scale multi-label video classification by addressing limitations in video representation learning, label independence assumptions, and model ensemble strategies.
• To enhance video representation through residual connections and hypercolumns, improving performance beyond mean-pooled features.
• To model complex label relationships (e.g., co-occurrence, exclusion) via a novel latent concept learning layer, moving beyond independent label assumptions.
• To design a systematic ensemble strategy using temporal segment augmentation and leave-one-out weighting to combine heterogeneous video-level and frame-level models effectively.
• To validate the superiority of attention-based pooling over RNNs for frame-level modeling and demonstrate the importance of complementary model types in ensembles.

Proposed method

• Proposes a mixture-of-residual-expert (MoRE) and mixture-of-hypercolumn-expert (MoHCE) model to refine video-level and frame-level representations using residual connections and multi-scale feature fusion.
• Introduces a delayed-start layer for latent concept learning that captures underlying relationships among concepts, improving multi-label classification beyond independent label predictions.
• Applies temporal segment data augmentation by splitting videos into segments and training on them, enhancing robustness and generalization of video-level models.
• Employs a leave-one-out ensemble strategy to determine optimal fusion weights for 64 heterogeneous models (42 video-level, 22 frame-level), avoiding overfitting and improving generalization.
• Uses attention-based pooling for frame-level modeling, which outperforms recurrent networks like LSTM and Bi-LSTM on the YouTube-8M dataset.
• Combines video-level models (mean-pooled features) and frame-level models in the ensemble, leveraging their complementary strengths: robustness to noise vs. localization of key shots.

Experimental results

Research questions

• RQ1 Can refined video representation learning with residual connections and hypercolumns improve performance on large-scale multi-label video classification beyond mean-pooled features?
• RQ2 Does learning latent concepts that model label dependencies (e.g., co-occurrence, exclusion) lead to better multi-label classification performance than assuming independent labels?
• RQ3 Can temporal segment data augmentation enhance the generalization of video-level models in a large-scale video tagging task?
• RQ4 Does a weighted ensemble of heterogeneous models (video-level vs. frame-level) outperform individual models or homogeneous ensembles?
• RQ5 Why do some high-performing individual models contribute less to the ensemble than expected, and how can model complementarity be systematically leveraged?

Key findings

• The proposed MoRE and MoHCE models significantly outperform baseline models, with the MoHCE model achieving 82.47% GAP on frame-level features.
• Latent concept learning via the delayed-start layer improves multi-label classification by capturing underlying label relationships, contributing to higher performance than independent label prediction.
• Attention-based pooling consistently outperformed RNN-based models (e.g., Bi-LSTM, LSTM with zoneout) on frame-level features, with the best frame-level model (NetVLAD + LC, k=1) achieving 82.47% GAP.
• The final ensemble of 34 models (18 video-level, 16 frame-level) achieved 84.68% GAP on the validation split and 84.66% on the official test set, demonstrating state-of-the-art performance.
• Leave-one-out analysis revealed that video-level models contributed 54% of the ensemble weight, indicating their strong and stable role, while frame-level models provided critical complementarity despite higher computational cost.
• The best single model was not the most important in the ensemble, suggesting that model diversity and complementarity matter more than individual performance, especially for hard examples.