[Paper Review] Delving Deeper into Convolutional Networks for Learning Video Representations

Nicolas Ballas, Li Yao+2 more|PolyPublie (École Polytechnique de Montréal)|Nov 19, 2015

Human Pose and Action Recognition32 references148 citations

TL;DR

This paper proposes a novel Recurrent Convolutional Network (RCN) that leverages multi-level visual percepts—activations from all layers of a pre-trained ImageNet CNN—combined with a convolutional GRU to model spatio-temporal features in videos. By applying a GRU with convolutional connectivity to low-level percepts, the method captures fine-grained motion patterns while reducing parameter count, achieving state-of-the-art performance on YouTube2Text video captioning and a 3.4% gain on UCF101 action recognition without 3D-CNN features.

ABSTRACT

We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset. While high-level percepts contain highly discriminative information, they tend to have a low-spatial resolution. Low-level percepts, on the other hand, preserve a higher spatial resolution from which we can model finer motion patterns. Using low-level percepts can leads to high-dimensionality video representations. To mitigate this effect and control the model number of parameters, we introduce a variant of the GRU model that leverages the convolution operations to enforce sparse connectivity of the model units and share parameters across the input spatial locations. We empirically validate our approach on both Human Action Recognition and Video Captioning tasks. In particular, we achieve results equivalent to state-of-art on the YouTube2Text dataset using a simpler text-decoder model and without extra 3D CNN features.

Motivation & Objective

• To address the limitation of existing RCNs that rely only on high-level CNN features, which discard fine spatial and temporal motion details.
• To improve video representation learning by exploiting visual percepts from multiple spatial resolutions in a deep CNN.
• To reduce model complexity when processing high-dimensional percepts by introducing convolutional connectivity in the GRU architecture.
• To demonstrate improved performance on video understanding tasks using a simpler decoder and without additional 3D-CNN features.

Proposed method

• The method extracts visual percepts from all layers of a pre-trained ImageNet CNN, using both low-level (high-resolution) and high-level (discriminative) features.
• A novel convolutional GRU variant replaces fully connected operations with convolutional ones, enforcing local connectivity and parameter sharing across spatial locations.
• The GRU processes percepts from each CNN layer independently, modeling temporal dynamics while preserving spatial topology.
• The model uses bidirectional GRUs to capture long-range temporal dependencies in video sequences.
• The final video representation is formed by concatenating the final hidden states of GRUs across all percept levels.
• The approach is trained end-to-end using cross-entropy loss with early stopping based on validation NLL.

Experimental results

Research questions

• RQ1 Can modeling temporal variation from low-level percepts improve video representation learning compared to relying only on high-level features?
• RQ2 How can the high-dimensional nature of low-level percepts be mitigated without losing spatial and temporal information?
• RQ3 Does a convolutional GRU architecture that enforces locality and parameter sharing outperform standard RNNs on video tasks?
• RQ4 Can this multi-level percept approach achieve state-of-the-art performance on video captioning without using 3D-CNN features?
• RQ5 Is the proposed method effective for both action recognition and video captioning with a unified, simpler decoder architecture?

Key findings

• The proposed method achieved a 10% relative improvement in BLEU score on the YouTube2Text video captioning benchmark compared to a VGG-16 encoder baseline.
• The model reached state-of-the-art performance on YouTube2Text using a simpler decoder and without any 3D-CNN features, outperforming models that use complex attention mechanisms or additional 3D-CNN encoders.
• On the UCF101 action recognition task, the method achieved a 3.4% absolute gain over the baseline model that used only top-layer percepts.
• The bidirectional GRU-RCN encoder outperformed the VGG-16 encoder across all metrics (BLEU, METEOR, CIDEr), with a 10% relative gain in BLEU.
• The combination of GoogleNet and the proposed GRU-RCN encoder achieved a BLEU score of 0.4963, surpassing models using C3D or hierarchical RNN decoders.
• The ablation study confirms that multi-level percept modeling is essential, as using only high-level percepts leads to suboptimal performance.