[Paper Review] Combining the Silhouette and Skeleton Data for Gait Recognition

Likai Wang, Ruize Han+1 more|arXiv (Cornell University)|Feb 22, 2022

Gait Recognition and Analysis4 citations

TL;DR

This paper proposes a two-branch deep learning framework that fuses silhouettes (via CNN) and skeleton data (via GCN) for robust gait recognition. It introduces a fully connected graph convolution operator to capture long-range joint dependencies and an STC-Att module for joint spatial, temporal, and channel-wise attention, achieving state-of-the-art performance, especially under clothing variation, with 91.0% mean accuracy on CASIA-B and 90.7% on OUMVLP.

ABSTRACT

Gait recognition, a long-distance biometric technology, has aroused intense interest recently. Currently, the two dominant gait recognition works are appearance-based and model-based, which extract features from silhouettes and skeletons, respectively. However, appearance-based methods are greatly affected by clothes-changing and carrying conditions, while model-based methods are limited by the accuracy of pose estimation. To tackle this challenge, a simple yet effective two-branch network is proposed in this paper, which contains a CNN-based branch taking silhouettes as input and a GCN-based branch taking skeletons as input. In addition, for better gait representation in the GCN-based branch, we present a fully connected graph convolution operator to integrate multi-scale graph convolutions and alleviate the dependence on natural joint connections. Also, we deploy a multi-dimension attention module named STC-Att to learn spatial, temporal and channel-wise attention simultaneously. The experimental results on CASIA-B and OUMVLP show that our method achieves state-of-the-art performance in various conditions.

Motivation & Objective

• To address the limitations of appearance-based gait recognition, which is sensitive to clothing and carrying changes.
• To overcome the accuracy ceiling of model-based methods due to pose estimation errors and sparse skeleton data.
• To combine the complementary strengths of appearance and model-based features for improved robustness.
• To design a GCN-based module that effectively captures multi-scale and long-range joint dependencies without relying on predefined bone connections.
• To enhance feature representation through a multi-dimension attention mechanism that models spatial, temporal, and channel-wise correlations simultaneously.

Proposed method

• A two-branch neural network architecture is used: one branch processes silhouettes with a CNN (using GaitPart as the backbone), and the other processes skeleton sequences with a GCN-based model.
• A fully connected graph convolution operator is proposed to replace multiple adjacency matrices, enabling adaptive integration of multi-scale joint dependencies without relying on natural joint connections.
• The STC-Att module learns spatial, temporal, and channel-wise attention in a single module, enhancing discriminative feature learning in the GCN branch.
• Features from both branches are concatenated after balancing with a learnable weight ratio λ to fuse appearance and structural representations.
• Global average pooling and a softmax classifier are applied to the fused features for final classification.
• The model is trained end-to-end on CASIA-B and OUMVLP datasets with standard cross-entropy loss.

Experimental results

Research questions

• RQ1 Can a two-branch network that fuses silhouettes and skeletons outperform existing state-of-the-art gait recognition methods under diverse conditions?
• RQ2 How effective is a fully connected graph convolution operator in modeling long-range joint dependencies compared to traditional GCNs with fixed adjacency matrices?
• RQ3 To what extent does the STC-Att module improve gait representation by jointly modeling spatial, temporal, and channel-wise attention?
• RQ4 Does the fusion of appearance and model-based features significantly enhance robustness to clothing and viewpoint variations?
• RQ5 What optimal fusion ratio λ balances the contributions of the two-branch features for maximum recognition accuracy across varying conditions?

Key findings

• The proposed method achieves a mean accuracy of 91.0% on CASIA-B and 90.7% on OUMVLP, outperforming all prior state-of-the-art methods.
• On the CASIA-B CL (clothing-changing) condition, the method achieves 92.1% accuracy, a 9.1% improvement over GaitGL, demonstrating strong robustness to clothing variation.
• The ablation study confirms that the fully connected graph convolution improves accuracy by 1.5% on average (from 89.8% to 91.3% on NM condition), validating its effectiveness.
• The STC-Att module further boosts performance by 0.5% on average (from 91.3% to 91.8% on NM), showing its value in feature refinement.
• The optimal fusion ratio λ = 400 achieves the highest mean accuracy (94.74%) across all conditions, confirming the benefit of balanced feature fusion.
• The two-branch network with λ = 400 achieves 97.7% on NM, 93.8% on BG, and 92.7% on CL, significantly outperforming individual branches (e.g., 91.8% for model-based only).