[Paper Review] 1st place solution for AVA-Kinetics Crossover in AcitivityNet Challenge 2020

Siyu Chen, Junting Pan+7 more|arXiv (Cornell University)|Jun 16, 2020

Human Pose and Action Recognition16 references4 citations

TL;DR

This paper presents a 1st place solution for the AVA-Kinetics Crossover track in ActivityNet Challenge 2020, introducing the Actor-Context-Actor Relation Network (ACAR-Net) that models high-order spatial-temporal relations between actors and scene context via a location-wise attention mechanism. The method achieves 39.62 mAP on the test set, significantly outperforming other entries by explicitly leveraging first-order actor-context relations and their higher-order interactions through a weakly-supervised relational reasoning block.

ABSTRACT

This technical report introduces our winning solution to the spatio-temporal action localization track, AVA-Kinetics Crossover, in ActivityNet Challenge 2020. Our entry is mainly based on Actor-Context-Actor Relation Network. We describe technical details for the new AVA-Kinetics dataset, together with some experimental results. Without any bells and whistles, we achieved 39.62 mAP on the test set of AVA-Kinetics, which outperforms other entries by a large margin. Code will be available at: https://github.com/Siyu-C/ACAR-Net.

Motivation & Objective

• To improve spatio-temporal action localization in the AVA-Kinetics Crossover dataset by modeling complex, higher-order relationships between actors and scene context.
• To develop an efficient and effective method for modeling indirect, high-order relations among actors via shared spatial context, which are often ignored by prior approaches.
• To leverage large-scale Kinetics data to improve generalization and performance on the AVA-Kinetics benchmark, despite domain shift and detection quality limitations.
• To design a weakly-supervised relational reasoning module that requires only action labels for supervision, avoiding costly human-annotated relation data.
• To achieve state-of-the-art performance using a simple yet powerful architecture that combines feature banks, relation modeling, and multi-scale inference.

Proposed method

• The framework uses a video backbone (e.g., I3D or SlowFast) and a person detector (e.g., Faster R-CNN) to extract spatio-temporal features and generate RoI proposals from the key frame.
• First-order actor-context relations are computed by concatenating each actor’s feature with the spatial grid of the video feature map, followed by convolutional processing to produce relation maps.
• A High-order Relation Reasoning Operator (HR²O) is introduced to model relations between first-order actor-context features at the same spatial location, using a modified, convolutional non-local block with layer normalization and dropout.
• The HR²O computes attention weights between different actors’ first-order relations at each spatial location, aggregating context features via weighted sum to produce high-order relation features.
• An Actor-Context Feature Bank (ACFB) is constructed from short clips to store and reuse first-order relation features over time, enhancing long-range temporal modeling.
• Final predictions are made by combining the RoI feature, high-order relation feature (via average pooling), and a classification head, with ensembling across multiple models and backbones.

Experimental results

Research questions

• RQ1 Can modeling high-order relations between actor-context features at the same spatial location improve spatio-temporal action localization performance?
• RQ2 How effective is a weakly-supervised, attention-based relational reasoning module in capturing indirect, higher-order dependencies without explicit relation annotations?
• RQ3 To what extent does pre-training on Kinetics-700 and fine-tuning on AVA-Kinetics improve generalization and mAP on the target benchmark?
• RQ4 How does the quality of person detection affect final action localization performance, especially when using model-generated boxes instead of ground truth?
• RQ5 Can a feature bank of first-order relations over time improve long-range temporal modeling and boost mAP in action localization?

Key findings

• The proposed ACAR-Net achieved 39.62 mAP on the AVA-Kinetics test set, outperforming all other entries in the ActivityNet Challenge 2020 by a large margin.
• Adding the ACAR-Net head to a simple linear classifier head increased mAP by 1.6 points, demonstrating the effectiveness of high-order relation modeling.
• Integrating the Actor-Context Feature Bank (ACFB) head further boosted mAP by 2.86 points over the baseline, highlighting the benefit of long-term feature memory.
• Using a more powerful SlowFast R152 backbone improved mAP by 0.54 points over R101, with minimal drop (-0.13 mAP) on the test set, indicating strong generalization.
• Training on AVA-Kinetics with Kinetics data increased mAP by approximately 2 points compared to training on AVA alone, showing the value of data mixing.
• Despite achieving 95.8 AP@50 on AVA person detection, the model still had an 8.1 mAP gap to ground-truth performance, indicating detection quality remains a bottleneck.