[Paper Review] TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting

Huazhang Hu, Sixun Dong+4 more|arXiv (Cornell University)|Apr 3, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes TransRAC, a transformer-based method for repetitive action counting that encodes multi-scale temporal correlations to improve performance on long, realistic videos with interruptions or inconsistent cycles. It introduces a new large-scale dataset, RepCount, with fine-grained cycle annotations, and uses density map regression for interpretable period prediction, achieving state-of-the-art results across datasets, including zero-shot generalization on unseen data.

ABSTRACT

Counting repetitive actions are widely seen in human activities such as physical exercise. Existing methods focus on performing repetitive action counting in short videos, which is tough for dealing with longer videos in more realistic scenarios. In the data-driven era, the degradation of such generalization capability is mainly attributed to the lack of long video datasets. To complement this margin, we introduce a new large-scale repetitive action counting dataset covering a wide variety of video lengths, along with more realistic situations where action interruption or action inconsistencies occur in the video. Besides, we also provide a fine-grained annotation of the action cycles instead of just counting annotation along with a numerical value. Such a dataset contains 1,451 videos with about 20,000 annotations, which is more challenging. For repetitive action counting towards more realistic scenarios, we further propose encoding multi-scale temporal correlation with transformers that can take into account both performance and efficiency. Furthermore, with the help of fine-grained annotation of action cycles, we propose a density map regression-based method to predict the action period, which yields better performance with sufficient interpretability. Our proposed method outperforms state-of-the-art methods on all datasets and also achieves better performance on the unseen dataset without fine-tuning. The dataset and code are available.

Motivation & Objective

• To address the limitations of existing repetitive action counting methods in long, realistic videos with interruptions or inconsistent action cycles.
• To develop a more interpretable and accurate model by introducing fine-grained cycle-level annotations instead of only numerical counts.
• To create a large-scale, realistic dataset (RepCount) covering diverse video lengths and anomalies to better evaluate and train models.
• To balance model performance and efficiency by leveraging multi-scale temporal correlation with transformers.
• To enable zero-shot generalization to unseen datasets through improved temporal modeling and architecture design.

Proposed method

• The method employs a multi-scale temporal correlation encoder using self-attention mechanisms to model long-range dependencies across different temporal scales.
• It introduces a density map regression head to predict the action period, enabling more interpretable and accurate localization of repetitive cycles compared to standard classification heads.
• The model uses a multi-column architecture with different receptive fields to capture temporal patterns at varying scales, fusing features for improved robustness.
• The self-attention mechanism computes correlation matrices dynamically, outperforming fixed similarity functions like TSM (temporal self-similarity matrix).
• The architecture is trained end-to-end on the RepCount dataset, which includes 1,451 videos and ~20,000 fine-grained cycle annotations.
• The method is evaluated using MAE (Mean Absolute Error) and OBO (Overall Best Overlap) metrics, demonstrating superior performance and zero-shot generalization.

Experimental results

Research questions

• RQ1 Can a transformer-based model effectively capture multi-scale temporal correlations in long, realistic videos for repetitive action counting?
• RQ2 Does fine-grained cycle-level annotation improve model interpretability and performance compared to only numerical counts?
• RQ3 Can a density map regression head outperform standard classification heads in predicting action periods for repetitive actions?
• RQ4 How does multi-scale modeling enhance performance and robustness in the presence of action interruptions or inconsistent cycles?
• RQ5 Can the proposed method generalize to unseen datasets without fine-tuning, indicating improved generalization capability?

Key findings

• TransRAC achieves a MAE of 0.4431 and OBO of 0.2913 on the RepCount part-A test set, outperforming state-of-the-art methods.
• The use of self-attention for correlation matrix computation improves performance over TSM-based similarity, reducing MAE by 0.1247.
• Replacing classification heads with density map regressors significantly improves performance, reducing MAE from 0.9950 (ResNet+CLS) to 0.6905 (ResNet+DM).
• Multi-scale modeling (Ours (Multi)) achieves the best performance (MAE: 0.4431), outperforming single-scale variants.
• The model generalizes well to unseen datasets without fine-tuning, demonstrating strong zero-shot generalization capability.
• The proposed RepCount dataset, with 1,451 videos and ~20,000 fine-grained cycle annotations, provides a more realistic and challenging benchmark for repetitive action counting.