[Paper Review] Context-aware and Scale-insensitive Temporal Repetition Counting

Huaidong Zhang, Xuemiao Xu+2 more|arXiv (Cornell University)|May 18, 2020

Human Pose and Action Recognition35 references4 citations

TL;DR

This paper proposes a context-aware and scale-insensitive framework for temporal repetition counting that addresses variable cycle lengths in real-world repetitive actions. It uses a coarse-to-fine refinement strategy with a bidirectional regression network on double-cycle inputs to improve accuracy across diverse time-scales, achieving state-of-the-art performance on multiple benchmarks including the newly introduced UCFRep dataset with 526 videos.

ABSTRACT

Temporal repetition counting aims to estimate the number of cycles of a given repetitive action. Existing deep learning methods assume repetitive actions are performed in a fixed time-scale, which is invalid for the complex repetitive actions in real life. In this paper, we tailor a context-aware and scale-insensitive framework, to tackle the challenges in repetition counting caused by the unknown and diverse cycle-lengths. Our approach combines two key insights: (1) Cycle lengths from different actions are unpredictable that require large-scale searching, but, once a coarse cycle length is determined, the variety between repetitions can be overcome by regression. (2) Determining the cycle length cannot only rely on a short fragment of video but a contextual understanding. The first point is implemented by a coarse-to-fine cycle refinement method. It avoids the heavy computation of exhaustively searching all the cycle lengths in the video, and, instead, it propagates the coarse prediction for further refinement in a hierarchical manner. We secondly propose a bidirectional cycle length estimation method for a context-aware prediction. It is a regression network that takes two consecutive coarse cycles as input, and predicts the locations of the previous and next repetitive cycles. To benefit the training and evaluation of temporal repetition counting area, we construct a new and largest benchmark, which contains 526 videos with diverse repetitive actions. Extensive experiments show that the proposed network trained on a single dataset outperforms state-of-the-art methods on several benchmarks, indicating that the proposed framework is general enough to capture repetition patterns across domains.

Motivation & Objective

• To address the challenge of variable and unpredictable cycle lengths in real-world repetitive actions, which existing methods fail to handle due to fixed-time-scale assumptions.
• To improve accuracy in repetition counting by incorporating contextual understanding across multiple action cycles, rather than relying on isolated segments.
• To reduce computational cost in cycle length search by using a hierarchical, coarse-to-fine refinement strategy instead of exhaustive global search.
• To establish a new, large-scale benchmark for temporal repetition counting to support future research.
• To demonstrate generalization across domains by training on a single dataset and outperforming SOTA on multiple benchmarks.

Proposed method

• A coarse-to-fine cycle refinement strategy is proposed, where initial cycle length estimates are made on local video clips and then propagated and refined across the entire video to reduce computational cost.
• A bidirectional regression network takes two consecutive action cycles (double-cycle) as input to predict the locations of both the previous and next repetitions, enabling context-aware estimation.
• The network uses a multi-anchor regression head to predict cycle lengths across a wide range of time-scales, improving robustness to diverse cycle variations.
• The framework integrates contextual features from both past and future repetitions to avoid miscounting in actions with similar sub-motions, such as alternating arm movements.
• A new benchmark, UCFRep, is constructed by annotating 526 videos from UCF101 to support training and evaluation in repetition counting.
• The method is trained end-to-end and evaluated using MAE and OBOA metrics across multiple datasets, including QUVA Repetition and YTsegments.

Experimental results

Research questions

• RQ1 Can a coarse-to-fine refinement strategy effectively reduce computational cost while maintaining accuracy in cycle length estimation for variable repetitive actions?
• RQ2 How does incorporating bidirectional context from two consecutive repetitions improve cycle length prediction compared to single-sequence modeling?
• RQ3 To what extent does multi-anchor regression enhance robustness to diverse and unpredictable cycle lengths?
• RQ4 Can a model trained on a single dataset generalize well across different benchmarks and action classes?
• RQ5 How does the proposed framework perform under varying time-scales, such as 1x, 2x, and 4x cycle length changes?

Key findings

• The proposed method achieves a mean absolute error (MAE) of 0.147 ± 0.243 on the UCFRep validation set, outperforming previous SOTA methods.
• The ablation study shows that using double time-scales (Free) with multi-anchors (mAnchor) yields the best performance (MAE: 0.147 ± 0.243, OBOA: 0.79), confirming the effectiveness of context and multi-scale prediction.
• The coarse-to-fine refinement process improves accuracy with each stage, with stage 5 (74 iterations) selected as the optimal trade-off between speed and performance.
• The model generalizes well across action classes, with low standard deviation in MAE (0.243 overall, 0.062 for Biking), indicating robustness to class-specific variations.
• The method achieves 8.0% OBOA on 4x cycle length variations, demonstrating strong scale-insensitivity across diverse time-scales.
• The model outperforms state-of-the-art methods on three benchmarks, including QUVA Repetition and YTsegments, confirming its generalization capability.