[Paper Review] Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation

Shuai Cheng Li, Feng Zhang+5 more|arXiv (Cornell University)|Jun 21, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes PRSA-Net, a Pyramid Region-based Slot Attention Network that enhances temporal action proposal generation by learning action-aware representations through a novel PRSlot module. Unlike similarity-based attention, PRSlot uses region-based attention and a parallel pyramid strategy to model local context more effectively, achieving state-of-the-art performance with 56.12% AR@100 and 58.7% mAP@0.5 on THUMOS1.3.

ABSTRACT

It has been found that temporal action proposal generation, which aims to discover the temporal action instances within the range of the start and end frames in the untrimmed videos, can largely benefit from proper temporal and semantic context exploitation. The latest efforts were dedicated to considering the temporal context and similarity-based semantic contexts through self-attention modules. However, they still suffer from cluttered background information and limited contextual feature learning. In this paper, we propose a novel Pyramid Region-based Slot Attention (PRSlot) module to address these issues. Instead of using the similarity computation, our PRSlot module directly learns the local relations in an encoder-decoder manner and generates the representation of a local region enhanced based on the attention over input features called extit{slot}. Specifically, upon the input snippet-level features, PRSlot module takes the target snippet as extit{query}, its surrounding region as extit{key} and then generates slot representations for each extit{query-key} slot by aggregating the local snippet context with a parallel pyramid strategy. Based on PRSlot modules, we present a novel Pyramid Region-based Slot Attention Network termed PRSA-Net to learn a unified visual representation with rich temporal and semantic context for better proposal generation. Extensive experiments are conducted on two widely adopted THUMOS14 and ActivityNet-1.3 benchmarks. Our PRSA-Net outperforms other state-of-the-art methods. In particular, we improve the AR@100 from the previous best 50.67% to 56.12% for proposal generation and raise the mAP under 0.5 tIoU from 51.9\% to 58.7\% for action detection on THUMOS14. extit{Code is available at} \url{https://github.com/handhand123/PRSA-Net}

Motivation & Objective

• To address the limitations of existing methods in capturing rich temporal and semantic context for temporal action proposal generation.
• To reduce sensitivity to cluttered background content in untrimmed videos, which degrades proposal quality.
• To improve proposal quality by modeling local, action-relevant context more effectively than pairwise similarity-based attention.
• To enhance feature representation learning through a structured, region-aware attention mechanism that focuses on local snippet contexts.
• To achieve state-of-the-art performance in both proposal generation and downstream action detection tasks.

Proposed method

• Proposes a Pyramid Region-based Slot Attention (PRSlot) module that learns object-centric representations by attending to local regions around a target snippet.
• Introduces Region-based Attention (RA) that directly estimates attention scores using an encoder-decoder architecture, restricting interactions to local spatial regions.
• Employs a parallel pyramid strategy to update slot representations across multiple scales simultaneously, avoiding iterative refinement and improving efficiency.
• Uses snippet-level features as input and generates enhanced slot representations that encode both local context and semantic relevance.
• Integrates PRSlot modules into a unified network (PRSA-Net) to produce boundary scores and proposal confidence scores end-to-end.
• Applies the PRSA-Net in a two-stage framework: first generating proposals, then classifying them using external or proposal-level classifiers.

Experimental results

Research questions

• RQ1 Can a region-based attention mechanism outperform similarity-based attention in modeling local context for temporal action proposal generation?
• RQ2 How does multi-scale, parallel pyramid slot updating improve feature representation compared to sequential or single-scale methods?
• RQ3 To what extent can PRSA-Net reduce the impact of background clutter and improve proposal quality in untrimmed videos?
• RQ4 Does the proposed PRSlot module lead to significant gains in both proposal generation and downstream action detection?
• RQ5 How does the model perform across different IoU thresholds and benchmarks such as THUMOS14 and ActivityNet-1.3?

Key findings

• PRSA-Net achieves 56.12% AR@100 on THUMOS1.3, improving from the previous SOTA of 50.67%.
• On THUMOS1.3, the mAP@0.5 for action detection increases from 45.4% to 55.0% when using the Unet classifier, and reaches 58.7% with the P-GCN proposal-level classifier.
• On ActivityNet-1.3, PRSA-Net achieves 52.37% mAP@0.5, outperforming previous SOTA methods by a significant margin.
• The ablation study confirms that the multi-scale pyramid strategy with {4,8} regions yields optimal performance, balancing accuracy and model complexity.
• Using two iterations in the PRSlot module yields the best performance, with degradation observed at three iterations.
• Qualitative results show that PRSA-Net successfully detects action boundaries even in videos with complex backgrounds and multiple overlapping actions.