[Paper Review] Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks

Pichao Wang, Wanqing Li+2 more|arXiv (Cornell University)|Dec 30, 2016

Human Pose and Action Recognition8 citations

TL;DR

This paper proposes Joint Trajectory Maps (JTM), a method that encodes 3D skeleton sequences into three 2D color images representing joint trajectories in orthogonal planes, enabling fine-tuning of pre-trained ConvNets for human action recognition. The approach achieves state-of-the-art performance on four benchmark datasets, including NTU RGB+D, by leveraging complementary spatial-temporal patterns and score fusion.

ABSTRACT

Convolutional Neural Networks (ConvNets) have recently shown promising performance in many computer vision tasks, especially image-based recognition. How to effectively apply ConvNets to sequence-based data is still an open problem. This paper proposes an effective yet simple method to represent spatio-temporal information carried in $3D$ skeleton sequences into three $2D$ images by encoding the joint trajectories and their dynamics into color distribution in the images, referred to as Joint Trajectory Maps (JTM), and adopts ConvNets to learn the discriminative features for human action recognition. Such an image-based representation enables us to fine-tune existing ConvNets models for the classification of skeleton sequences without training the networks afresh. The three JTMs are generated in three orthogonal planes and provide complimentary information to each other. The final recognition is further improved through multiply score fusion of the three JTMs. The proposed method was evaluated on four public benchmark datasets, the large NTU RGB+D Dataset, MSRC-12 Kinect Gesture Dataset (MSRC-12), G3D Dataset and UTD Multimodal Human Action Dataset (UTD-MHAD) and achieved the state-of-the-art results.

Motivation & Objective

• To address the challenge of effectively representing spatio-temporal information in 3D skeleton sequences for deep learning-based action recognition.
• To enable the use of pre-trained ConvNets on image data for skeleton sequence classification without retraining from scratch.
• To improve recognition accuracy through complementary representation in three orthogonal planes and late score fusion.
• To enhance robustness to viewpoint variations via data augmentation using rotated skeleton views.
• To demonstrate the effectiveness of image-based encoding over traditional sequence modeling or hand-crafted features.

Proposed method

• Joint Trajectory Maps (JTM) are generated by projecting 3D joint trajectories onto three orthogonal planes (e.g., XY, XZ, YZ), encoding position and motion dynamics into color intensity.
• Each JTM uses color encoding to represent the trajectory of each joint over time, creating texture-like patterns that preserve spatial and temporal structure.
• The method applies rotation augmentation to skeleton data, generating multiple views to simulate cross-view data and improve generalization.
• Pre-trained ImageNet ConvNets are fine-tuned on the three JTM images to extract discriminative features for action classification.
• Final predictions are fused using late score averaging across the three JTM views to improve robustness and accuracy.
• The approach avoids training deep networks from scratch by leveraging transfer learning on image-based representations.

Experimental results

Research questions

• RQ1 Can 3D skeleton sequences be effectively transformed into 2D image-like representations that preserve spatio-temporal dynamics for ConvNet processing?
• RQ2 Does using three orthogonal JTM views provide complementary information that improves action recognition beyond single-view representations?
• RQ3 Can pre-trained ConvNets trained on natural images be effectively fine-tuned for action recognition on skeleton sequences using JTM input?
• RQ4 How does data augmentation via skeleton rotation affect recognition performance and viewpoint generalization?
• RQ5 Does fusing scores from multiple JTM views lead to consistent performance gains across diverse datasets?

Key findings

• The proposed method achieved state-of-the-art accuracy on the large-scale NTU RGB+D dataset, outperforming prior methods using similar input modalities.
• On the G3D dataset, the method achieved 95.15% recognition accuracy when fusing nine views (θ, ψ ∈ {-22.5°, 0°, 22.5°}), with no significant gain from using 25 views.
• The natural coordinate system (θ=0°, ψ=0°) produced the best single-view accuracy (95.12%) on G3D, though variations across views were minimal.
• Fusing three orthogonal JTMs from the natural coordinate system improved recognition accuracy over individual views, demonstrating complementary information gain.
• Rotation augmentation effectively improved generalization, especially for cross-view action recognition, without requiring explicit viewpoint modeling.
• The method achieved strong performance across four diverse datasets—NTU RGB+D, MSRC-12, G3D, and UTD-MHAD—confirming its robustness and generalization capability.