[Paper Review] Modeling Spatio-Temporal Human Track Structure for Action Localization

Guilhem Chéron, Anton Osokin+2 more|arXiv (Cornell University)|Jun 28, 2018

Human Pose and Action Recognition59 references4 citations

TL;DR

This paper proposes RecLNet, a recurrent localization network using gated recurrent units (GRUs) to model temporal action structure on person tracks, improving spatio-temporal action localization. By processing appearance and optical flow features through two-stream GRUs and applying threshold-based temporal filtering, RecLNet achieves state-of-the-art performance on UCF101-24 and DALY, significantly enhancing temporal boundary detection over baseline CNNs.

ABSTRACT

This paper addresses spatio-temporal localization of human actions in video. In order to localize actions in time, we propose a recurrent localization network (RecLNet) designed to model the temporal structure of actions on the level of person tracks. Our model is trained to simultaneously recognize and localize action classes in time and is based on two layer gated recurrent units (GRU) applied separately to two streams, i.e. appearance and optical flow streams. When used together with state-of-the-art person detection and tracking, our model is shown to improve substantially spatio-temporal action localization in videos. The gain is shown to be mainly due to improved temporal localization. We evaluate our method on two recent datasets for spatio-temporal action localization, UCF101-24 and DALY, demonstrating a significant improvement of the state of the art.

Motivation & Objective

• To improve spatio-temporal action localization by modeling the temporal structure of actions at the level of individual person tracks.
• To address the weakness of existing methods in precise temporal boundary detection, which often leads to false positives and imprecise localization.
• To develop a recurrent neural network that jointly recognizes and localizes actions in time, leveraging both appearance and motion features.
• To demonstrate that recurrence on person tracks significantly improves temporal localization accuracy, especially where baseline methods fail.
• To show that the method is complementary to recent advances in spatial detection precision, with room for further improvement in spatial localization.

Proposed method

• RecLNet employs two-stream gated recurrent units (GRUs) to process appearance and optical flow features extracted from person bounding boxes in video sequences.
• Features are pooled from region proposals using Fast-RCNN and fed into separate GRU layers to model long-term temporal dependencies within each person track.
• A thresholding and filtering strategy is applied to the GRU outputs to generate final action detections with precise temporal boundaries.
• The model is trained end-to-end to jointly recognize action classes and localize their temporal intervals on a per-track basis.
• Temporal localization is optimized by filtering high-scoring responses and applying a threshold to suppress false positives.
• The method is evaluated using standard benchmarks UCF101-24 and DALY, with results compared against CNN-based baselines and state-of-the-art methods.

Experimental results

Research questions

• RQ1 Can modeling temporal structure on person tracks using recurrent networks improve temporal action localization accuracy?
• RQ2 Does the use of GRUs on appearance and optical flow streams lead to better detection of action boundaries compared to standard CNNs?
• RQ3 How does the proposed RecLNet method compare to state-of-the-art approaches in terms of spatio-temporal localization performance?
• RQ4 To what extent is the improvement due to better temporal localization rather than spatial detection?
• RQ5 Can the method be combined with recent advances in spatial detection to further improve overall performance?

Key findings

• RecLNet achieves state-of-the-art performance on the UCF101-24 dataset, significantly outperforming previous methods in spatio-temporal action localization.
• On the DALY dataset, RecLNet demonstrates substantial improvements over the CNN RGB+OF baseline, particularly in temporal boundary localization.
• Qualitative analysis shows that RecLNet scores are better aligned with ground-truth action boundaries than baseline methods, which often produce imprecise or noisy responses.
• The method reduces false positives, as evidenced by fewer high-scoring dashed lines in qualitative results, indicating more reliable detection.
• Temporal localization is identified as a key weakness in existing methods, and RecLNet effectively corrects this issue through recurrent modeling.
• The method is complementary to recent advances in spatial precision, suggesting that future improvements could come from combining better spatial detection with RecLNet's strong temporal modeling.