[Paper Review] Context-Aware RCNN: A Baseline for Action Detection in Videos

Jian-Chao Wu, Zhanghui Kuang+3 more|arXiv (Cornell University)|Jul 20, 2020

Human Pose and Action Recognition33 references4 citations

TL;DR

This paper proposes Context-Aware RCNN, a simple yet effective baseline for video action detection that improves performance by cropping and resizing actor regions to higher resolution before feature extraction with I3D, while fusing scene and long-term context. It achieves state-of-the-art mAP of 28.0% on AVA and 79.2% on JHMDB, outperforming prior methods despite using a standard ResNet-50 backbone.

ABSTRACT

Video action detection approaches usually conduct actor-centric action recognition over RoI-pooled features following the standard pipeline of Faster-RCNN. In this work, we first empirically find the recognition accuracy is highly correlated with the bounding box size of an actor, and thus higher resolution of actors contributes to better performance. However, video models require dense sampling in time to achieve accurate recognition. To fit in GPU memory, the frames to backbone network must be kept low-resolution, resulting in a coarse feature map in RoI-Pooling layer. Thus, we revisit RCNN for actor-centric action recognition via cropping and resizing image patches around actors before feature extraction with I3D deep network. Moreover, we found that expanding actor bounding boxes slightly and fusing the context features can further boost the performance. Consequently, we develop a surpringly effective baseline (Context-Aware RCNN) and it achieves new state-of-the-art results on two challenging action detection benchmarks of AVA and JHMDB. Our observations challenge the conventional wisdom of RoI-Pooling based pipeline and encourage researchers rethink the importance of resolution in actor-centric action recognition. Our approach can serve as a strong baseline for video action detection and is expected to inspire new ideas for this filed. The code is available at \url{https://github.com/MCG-NJU/CRCNN-Action}.

Motivation & Objective

• To investigate whether the standard RoI-Pooling pipeline in action detection loses discriminative spatial details due to low-resolution inputs.
• To address performance degradation in fine-grained action recognition caused by small actor bounding boxes.
• To develop a simple yet effective baseline for action detection that preserves local details and leverages contextual information.
• To challenge the conventional wisdom of using RoI-Pooling in actor-centric action recognition and propose a more effective alternative.

Proposed method

• Crop and resize actor bounding boxes from original video frames to a fixed high resolution before feeding into a 3D CNN (I3D) for feature extraction.
• Use a pre-trained person detector to generate actor proposals at key frames, then extract features from the cropped patches.
• Fuse scene features extracted from the full video clip and long-term features from a simplified non-local module to enhance context understanding.
• Aggregate actor features, scene features, and long-term features through concatenation for final action classification.
• Apply a simplified non-local block to capture long-range temporal dependencies, improving performance over standard pooling.
• Use average pooling for feature aggregation, which performs comparably to attention-based methods but with lower computational cost.

Experimental results

Research questions

• RQ1 Does the RoI-Pooling-based pipeline in action detection degrade performance due to loss of spatial detail in small actor regions?
• RQ2 Can improving input resolution for small actors through cropping and resizing enhance recognition accuracy in actor-centric action detection?
• RQ3 How effective is context modeling—specifically scene and long-term features—in improving action detection performance?
• RQ4 Can a simple RCNN-like pipeline with high-resolution actor crops outperform complex state-of-the-art models?

Key findings

• The RoI-Pooling-based pipeline suffers significant performance degradation when actor bounding boxes are small, with mAP dropping sharply as resolution decreases.
• Context-Aware RCNN achieves a new state-of-the-art mAP of 28.0% on the AVA dataset, surpassing the previous best method (LFB) by 2.2%.
• Even with a smaller ResNet-50 backbone, Context-Aware RCNN outperforms LFB with a deeper ResNet-101, demonstrating the effectiveness of high-resolution input.
• The addition of scene features improves mAP from 24.7% to 25.7%, and incorporating long-term features boosts it further to 28.0%.
• On the JHMDB dataset, the model achieves 79.2% mAP using only RGB frames, setting a new SOTA for this benchmark.
• The simplified non-local module outperforms standard attention mechanisms and achieves comparable results to more complex attention blocks.