[Paper Review] Action Recognition with Joint Attention on Multi-Level Deep Features

Jialin Wu, Gu Wang+2 more|arXiv (Cornell University)|Jul 9, 2016

Human Pose and Action Recognition17 references13 citations

TL;DR

This paper proposes a deep supervised neural network with joint attention on multi-level deep features for video action recognition, leveraging 3D-convolutional networks and a novel joint LSTM module to enhance temporal modeling and robustness. The model achieves state-of-the-art performance on UCF101 and HMDB51 using only RGB features, outperforming prior methods by up to 20.4% over baseline attention models.

ABSTRACT

We propose a novel deep supervised neural network for the task of action recognition in videos, which implicitly takes advantage of visual tracking and shares the robustness of both deep Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). In our method, a multi-branch model is proposed to suppress noise from background jitters. Specifically, we firstly extract multi-level deep features from deep CNNs and feed them into 3d-convolutional network. After that we feed those feature cubes into our novel joint LSTM module to predict labels and to generate attention regularization. We evaluate our model on two challenging datasets: UCF101 and HMDB51. The results show that our model achieves the state-of-art by only using convolutional features.

Motivation & Objective

• Address the challenge of capturing temporal dynamics in video action recognition while maintaining robustness to background clutter and intra-class variance.
• Overcome limitations of trajectory-based and single-branch feature extraction by integrating multi-level deep features from pre-trained CNNs.
• Improve performance through a novel attention regularization mechanism that enhances both 3D-convnet and joint LSTM modules.
• Design adaptive fusion mechanisms via two novel joint LSTM structures to effectively combine features from different levels of abstraction.

Proposed method

• Extract multi-level deep features from a pre-trained VGG-16 network (e.g., pool4 and conv5_3 layers) to capture both general and specific visual patterns.
• Feed the extracted features into a 3D-convolutional network to model spatiotemporal dependencies across video clips.
• Employ a novel joint LSTM module that jointly predicts action labels and generates attention maps for feature regularization.
• Introduce a soft supervised regularization term that guides attention learning and improves training convergence and generalization.
• Implement two fusion strategies—early and late fusion—within the joint LSTM to adaptively combine multi-level features.
• Train the entire model end-to-end using cross-entropy loss with attention regularization to enhance discriminative power.

Experimental results

Research questions

• RQ1 Can joint attention on multi-level deep features improve action recognition performance without relying on optical flow or complex hand-crafted features?
• RQ2 How does attention regularization enhance the learning of both 3D-convolutional and LSTM components in the model?
• RQ3 What is the impact of using multi-level features (e.g., from different VGG-16 layers) compared to single-level features?
• RQ4 How do early and late fusion strategies in the joint LSTM affect performance and feature integration?
• RQ5 Can the proposed model rival or surpass state-of-the-art methods using only RGB input, without additional modality fusion?

Key findings

• The proposed model achieves 90.6% top-1 accuracy on UCF101 and 61.7% on HMDB51, outperforming the baseline soft attention model by 20.4% in absolute accuracy.
• Using multi-branch features from different VGG-16 layers (e.g., pool4 and conv5_3) improves performance significantly, with late fusion reaching 61.7% on HMDB51.
• The 3D-convolutional network outperforms 2D-convnet and direct LSTM input by 4.4% and 4.6% respectively, demonstrating its effectiveness in capturing spatiotemporal patterns.
• Attention regularization leads to faster convergence and lower training loss, with the model stabilizing by the third epoch.
• The joint LSTM with late fusion architecture achieves better performance than early fusion, indicating improved feature fusion and independence.
• Qualitative attention maps show that the model successfully focuses on action-relevant regions (e.g., hands, face, limbs), aligning with human perception of key action cues.