[Paper Review] Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks

Alberto Montes, Amaia Salvador+2 more|arXiv (Cornell University)|Aug 29, 2016

Human Pose and Action Recognition11 references82 citations

TL;DR

This paper proposes a simple yet effective pipeline for temporal activity detection in untrimmed videos using 3D-CNN features (C3D) fed into an RNN with LSTM units to classify and localize activities. The method achieves a 0.5874 mAP on classification and 0.2237 mAP on detection in the ActivityNet Challenge 2016, with post-processing enhancing localization accuracy through smoothing and thresholding.

ABSTRACT

This thesis explore different approaches using Convolutional and Recurrent Neural Networks to classify and temporally localize activities on videos, furthermore an implementation to achieve it has been proposed. As the first step, features have been extracted from video frames using an state of the art 3D Convolutional Neural Network. This features are fed in a recurrent neural network that solves the activity classification and temporally location tasks in a simple and flexible way. Different architectures and configurations have been tested in order to achieve the best performance and learning of the video dataset provided. In addition it has been studied different kind of post processing over the trained network's output to achieve a better results on the temporally localization of activities on the videos. The results provided by the neural network developed in this thesis have been submitted to the ActivityNet Challenge 2016 of the CVPR, achieving competitive results using a simple and flexible architecture.

Motivation & Objective

• To address the challenge of classifying and temporally localizing activities in untrimmed videos, where videos are not pre-segmented.
• To develop a simple, end-to-end trainable framework that leverages both spatial and temporal features from video sequences.
• To improve detection performance through post-processing techniques that refine RNN output sequences.
• To achieve competitive results on the ActivityNet Challenge 2016 benchmark for both classification and temporal localization tasks.

Proposed method

• Extract 4096-dimensional C3D fc6 features from 16-frame clips of untrimmed videos, resized to 171×128 for input.
• Feed the sequence of C3D features into a stacked LSTM network with dropout (p=0.5) for sequence modeling and temporal dependency learning.
• Use a final softmax layer to output class probabilities for each 16-frame clip, with K+1 classes including a background class.
• Apply a mean filter (k=5) to smooth the predicted activity probabilities over time to reduce noise in the sequence.
• Apply a threshold γ to retain only clips with activity probability above γ, labeling them with the predicted class.
• Assign the final video-level class as the one with maximum average probability across all clips.

Experimental results

Research questions

• RQ1 Can a simple RNN-based architecture using pre-extracted C3D features achieve competitive performance in both video classification and temporal activity detection?
• RQ2 How does post-processing with smoothing and thresholding affect the localization accuracy of activity predictions?
• RQ3 What is the optimal RNN architecture (number of layers and cells) for balancing performance and generalization in untrimmed video activity detection?
• RQ4 How does class imbalance in the dataset affect model training, and what strategies can mitigate its impact?

Key findings

• The single-layer 512-LSTM configuration achieved the best classification mAP of 0.5938, outperforming deeper architectures due to reduced overfitting.
• The 3x1024-LSTM model achieved the highest Hit@3 score of 0.7437, indicating strong top-3 prediction accuracy.
• Post-processing with a mean filter (k=5) and threshold γ=0.2 improved detection mAP to 0.22513, the highest among tested configurations.
• The optimal post-processing parameters were γ=0.2 and k=5, yielding the best balance between precision and recall in localization.
• The model achieved a detection mAP of 0.2237 on the ActivityNet 2016 test set, demonstrating strong performance on the temporal localization task.
• The results show that even a simple pipeline combining C3D features with an RNN can achieve competitive results without end-to-end training.