[Paper Review] Learnable pooling with Context Gating for video classification

Antoine Miech, Ivan Laptev+1 more|arXiv (Cornell University)|Jun 21, 2017

Human Pose and Action RecognitionComputer Science48 references242 citations

TL;DR

The paper introduces Context Gating to reweight features and outputs, and explores learnable, clustering-based pooling methods (NetVLAD, NetFV, BoW, NetRVLAD) for scalable video classification, achieving state-of-the-art results on Youtube-8M V2.

ABSTRACT

Current methods for video analysis often extract frame-level features using pre-trained convolutional neural networks (CNNs). Such features are then aggregated over time e.g., by simple temporal averaging or more sophisticated recurrent neural networks such as long short-term memory (LSTM) or gated recurrent units (GRU). In this work we revise existing video representations and study alternative methods for temporal aggregation. We first explore clustering-based aggregation layers and propose a two-stream architecture aggregating audio and visual features. We then introduce a learnable non-linear unit, named Context Gating, aiming to model interdependencies among network activations. Our experimental results show the advantage of both improvements for the task of video classification. In particular, we evaluate our method on the large-scale multi-modal Youtube-8M v2 dataset and outperform all other methods in the Youtube 8M Large-Scale Video Understanding challenge.

Motivation & Objective

• Motivate better temporal aggregation for video classification beyond simple averaging and recurrent nets.
• Propose Context Gating to model interdependencies among activations and calibrate feature and label importance.
• Investigate clustering-based, differentiable pooling (NetVLAD, NetFV, BoW, NetRVLAD) as alternatives to LSTMs/GRUs.
• Show that combining audio and visual streams improves multi-modal video understanding.
• Demonstrate state-of-the-art performance on the Youtube-8M v2 dataset using learnable pooling and gating.

Proposed method

• Introduce a two-stream pooling architecture for visual and audio features.
• Propose Context Gating, a non-linear gating unit Y = sigma(WX + b) ∘ X to gate each feature dimension.
• Apply Context Gating after pooling and after the classifier to capture output-space priors.
• Adapt NetVLAD, NetFV, NetRVLAD, and NetBoW as differentiable, learnable pooling methods for temporal aggregation.
• Compare against LSTM/GRU baselines and simple average pooling under the same 1024-dimensional representation.
• Use Mixture-of-Experts (MoE) classifier followed by Context Gating in the final stage.

Experimental results

Research questions

• RQ1 Can learnable, clustering-based pooling schemes outperform recurrent models for large-scale video classification?
• RQ2 Does Context Gating improve both feature-level representations and output-space priors in multi-modal video tasks?
• RQ3 How do audio-visual fusion strategies interact with different pooling methods for Youtube-8M data?
• RQ4 What is the generalization behavior of these pooling methods with increasing training data?
• RQ5 Can the proposed methods achieve state-of-the-art results on a large-scale video understanding benchmark?

Key findings

• Learnable pooling methods (BoW, NetVLAD, NetFV, NetRVLAD) outperform average pooling and recurrent models in GAP on Youtube-8M v2.
• Context Gating consistently improves performance when applied to clustering-based pooling methods.
• Gated NetVLAD achieves 83.2% GAP, and Gated NetRVLAD achieves 83.1% GAP on the validation set (per Table I).
• Two-stream audio-visual fusion with late concatenation enhances clustering-based pooling performance.
• Context Gating provides a significant 0.8% GAP gain on NetVLAD with 128 clusters (ablation).
• Ensembling multiple diverse models yields the top performance, with a full ensemble of 25 models reaching 85.0% GAP on their setup.