[Paper Review] LocATe: End-to-end Localization of Actions in 3D with Transformers

Jiankai Sun, Bolei Zhou+2 more|arXiv (Cornell University)|Mar 21, 2022

Human Pose and Action Recognition4 citations

TL;DR

LocATe proposes an end-to-end transformer-based method for 3D Temporal Action Localization (3D-TAL) that jointly detects action spans and classifies actions using 3D human pose sequences. It introduces sparse self-attention and a set-prediction formulation to model long-range temporal dependencies, achieving state-of-the-art mAP of 93.2% on PKU-MMD and a new, more challenging benchmark BT20.

ABSTRACT

Understanding a person's behavior from their 3D motion is a fundamental problem in computer vision with many applications. An important component of this problem is 3D Temporal Action Localization (3D-TAL), which involves recognizing what actions a person is performing, and when. State-of-the-art 3D-TAL methods employ a two-stage approach in which the action span detection task and the action recognition task are implemented as a cascade. This approach, however, limits the possibility of error-correction. In contrast, we propose LocATe, an end-to-end approach that jointly localizes and recognizes actions in a 3D sequence. Further, unlike existing autoregressive models that focus on modeling the local context in a sequence, LocATe's transformer model is capable of capturing long-term correlations between actions in a sequence. Unlike transformer-based object-detection and classification models which consider image or patch features as input, the input in 3D-TAL is a long sequence of highly correlated frames. To handle the high-dimensional input, we implement an effective input representation, and overcome the diffuse attention across long time horizons by introducing sparse attention in the model. LocATe outperforms previous approaches on the existing PKU-MMD 3D-TAL benchmark (mAP=93.2%). Finally, we argue that benchmark datasets are most useful where there is clear room for performance improvement. To that end, we introduce a new, challenging, and more realistic benchmark dataset, BABEL-TAL-20 (BT20), where the performance of state-of-the-art methods is significantly worse. The dataset and code for the method will be available for research purposes.

Motivation & Objective

• Address the limitations of two-stage 3D-TAL methods that treat localization and recognition independently, leading to error propagation and lack of mutual refinement.
• Develop an end-to-end framework that jointly optimizes action localization and recognition using a single set of features.
• Overcome the challenges of modeling long-range temporal dependencies in high-dimensional 3D pose sequences using a transformer-based architecture.
• Introduce a new, realistic, and challenging benchmark dataset, BABEL-TAL-20 (BT20), to stimulate further progress in 3D-TAL beyond saturated performance on existing datasets.
• Demonstrate that raw 3D joint positions outperform 2D video features and action recognition backbone features as input representations for 3D-TAL.

Proposed method

• Formulate 3D-TAL as a set-prediction problem, where the model outputs a set of (start time, end time, action label) tuples, enabling modeling of simultaneous actions.
• Use a transformer encoder-decoder architecture with sparse self-attention to efficiently model long-range temporal correlations in 3D pose sequences.
• Input the model with 8-frame contiguous chunks of 3D joint positions, which are embedded and processed via self-attention to capture short- and long-term motion patterns.
• Implement a fully attentional design without relying on pre-trained 2D or action recognition features, using only 3D joint positions as input.
• Apply deformable attention mechanisms to reduce computational cost and improve attention focus across long sequences.
• Use optimal bipartite matching for evaluation and Non-Maximum Suppression (NMS) for final output refinement in downstream applications.

Experimental results

Research questions

• RQ1 Can an end-to-end transformer-based model jointly improve action localization and recognition in 3D-TAL compared to cascaded two-stage methods?
• RQ2 Does modeling long-range temporal dependencies in 3D pose sequences via sparse self-attention lead to better performance than autoregressive or two-stage baselines?
• RQ3 Is raw 3D joint position input more effective than 2D video features or features from pre-trained action recognition models for 3D-TAL?
• RQ4 To what extent does the performance of 3D-TAL models depend on class frequency versus intra-class variance and action complexity?
• RQ5 Can a new, more realistic benchmark dataset like BABEL-TAL-20 effectively stimulate progress in 3D-TAL by exposing limitations of current state-of-the-art methods?

Key findings

• LocATe achieves a state-of-the-art mAP of 93.2% on the PKU-MMD benchmark, significantly outperforming previous two-stage methods.
• On the newly introduced BABEL-TAL-20 (BT20) benchmark, state-of-the-art methods achieve substantially lower performance, indicating that BT20 is more challenging and realistic.
• The model successfully localizes and recognizes complex actions such as 'grasp something' and 'place something' even without explicit hand or finger joint data.
• The ablation study shows that 3D joint positions as input yield the best performance (mAP 23.4 on BT20), outperforming 2D features (14.5) and features from 2S-AGCN (21.3–21.4).
• Despite low recall for some actions like 'grasp something', the model predicts them frequently and correctly, suggesting strong temporal and full-body context modeling.
• The lack of clear correlation between class frequency and mAP indicates that intra-class variance and action complexity are major contributors to difficulty beyond sample scarcity.