[Paper Review] Skepxels: Spatio-temporal Image Representation of Human Skeleton Joints for Action Recognition

Jian Liu, Naveed Akhtar+1 more|arXiv (Cornell University)|Nov 16, 2017

Human Pose and Action Recognition53 references17 citations

TL;DR

This paper proposes Skepxels, a novel spatio-temporal image representation that transforms human skeleton joint sequences into flexible-dimensional images to enable effective CNN-based action recognition. By organizing joints into 2D grids using a unique distance metric and encoding both location and velocity, Skepxels capture rich spatio-temporal correlations, achieving state-of-the-art accuracy improvements of 4.4%, 5.7%, and 9.3% on NTU, NUCLA, and UTD-MHAD datasets respectively.

ABSTRACT

Human skeleton joints are popular for action analysis since they can be easily extracted from videos to discard background noises. However, current skeleton representations do not fully benefit from machine learning with CNNs. We propose "Skepxels" a spatio-temporal representation for skeleton sequences to fully exploit the "local" correlations between joints using the 2D convolution kernels of CNN. We transform skeleton videos into images of flexible dimensions using Skepxels and develop a CNN-based framework for effective human action recognition using the resulting images. Skepxels encode rich spatio-temporal information about the skeleton joints in the frames by maximizing a unique distance metric, defined collaboratively over the distinct joint arrangements used in the skeletal image. Moreover, they are flexible in encoding compound semantic notions such as location and speed of the joints. The proposed action recognition exploits the representation in a hierarchical manner by first capturing the micro-temporal relations between the skeleton joints with the Skepxels and then exploiting their macro-temporal relations by computing the Fourier Temporal Pyramids over the CNN features of the skeletal images. We extend the Inception-ResNet CNN architecture with the proposed method and improve the state-of-the-art accuracy by 4.4% on the large scale NTU human activity dataset. On the medium-sized N-UCLA and UTH-MHAD datasets, our method outperforms the existing results by 5.7% and 9.3% respectively.

Motivation & Objective

• To address the limited exploitation of CNNs in skeleton-based action recognition due to sparse, non-image-like joint data.
• To overcome the limitations of columnar or upsampled skeleton representations that distort semantics and introduce noise.
• To develop a flexible, learnable image representation that preserves spatio-temporal correlations between skeleton joints across frames.
• To enable hierarchical modeling of micro-temporal joint relations and macro-temporal action patterns using CNN features.
• To demonstrate that Skepxels generalize well to real-world RGB video skeletons, not just precise 3D skeleton data.

Proposed method

• Skepxels define a new atomic visual unit—skeleton picture element—by organizing joint positions from multiple frames into a 2D tensor using a unique distance metric over joint arrangements.
• The method constructs skeletal images of arbitrary dimensions by arranging Skepxels from different frames, preserving spatio-temporal dynamics.
• It encodes both joint location and velocity by incorporating temporal differences between consecutive frames into the image tensor.
• The framework uses a hierarchical CNN pipeline: first, Inception-ResNet extracts micro-temporal features from Skepxel images; then, Fourier Temporal Pyramids model macro-temporal dependencies across frames.
• The approach supports multiple CNN architectures (Inception-ResNet, GoogLeNet, ResNet) and enables model ensembling for further accuracy gains.
• The representation is robust to noisy skeleton data, as shown by comparable performance on RGB videos using DeeperCut-parsed joints.

Experimental results

Research questions

• RQ1 Can a CNN-based action recognition framework achieve state-of-the-art performance using only skeleton joint sequences if properly represented as images?
• RQ2 How does a 2D grid-based skeleton image representation compare to columnar or upsampled representations in capturing spatio-temporal correlations?
• RQ3 To what extent can Skepxels encode semantic notions like joint location and velocity within a single image tensor?
• RQ4 Does the proposed representation generalize to real-world RGB video data with imperfect skeleton extraction?
• RQ5 Can the hierarchical feature learning (micro- and macro-temporal) using Skepxels significantly improve recognition accuracy across diverse datasets?

Key findings

• The proposed Skepxel representation improves action recognition accuracy by 4.4% on the large-scale NTU Human Activity Dataset, setting a new state of the art.
• On the NUCLA dataset, the method achieves a 5.7% accuracy gain over existing methods, demonstrating strong generalization.
• On the UTD-MHAD dataset, the accuracy improves by 9.3%, highlighting the effectiveness of the representation on medium-sized, multimodal action data.
• The use of both location and velocity information in Skepxels (loc+vel) yields consistently higher performance than location-only encoding across all datasets and CNN architectures.
• The representation generalizes well to noisy skeleton data: with DeeperCut-parsed joints, the method achieves 92.3% accuracy on UTD-MHAD, comparable to results with precise 3D skeletons.
• Ensembling multiple CNNs (e.g., Inception-ResNet + ResNet) with Skepxels further boosts performance, reaching 98.6% accuracy on UTD-MHAD, confirming the scalability and robustness of the approach.