[Paper Review] Exploring Deep Models for Practical Gait Recognition

Chao Fan, Saihui Hou+2 more|arXiv (Cornell University)|Mar 6, 2023

Gait Recognition and Analysis17 citations

TL;DR

The paper demonstrates that deep CNN- and transformer-based gait models (DeepGaitV2 and SwinGait) significantly improve real-world gait recognition, emphasizes explicit temporal modeling, and resolves the dumb-patch issue in transformers for gait silhouettes.

ABSTRACT

Gait recognition is a rapidly advancing vision technique for person identification from a distance. Prior studies predominantly employed relatively shallow networks to extract subtle gait features, achieving impressive successes in constrained settings. Nevertheless, experiments revealed that existing methods mostly produce unsatisfactory results when applied to newly released real-world gait datasets. This paper presents a unified perspective to explore how to construct deep models for state-of-the-art outdoor gait recognition, including the classical CNN-based and emerging Transformer-based architectures. Specifically, we challenge the stereotype of shallow gait models and demonstrate the superiority of explicit temporal modeling and deep transformer structure for discriminative gait representation learning. Consequently, the proposed CNN-based DeepGaitV2 series and Transformer-based SwinGait series exhibit significant performance improvements on Gait3D and GREW. As for the constrained gait datasets, the DeepGaitV2 series also reaches a new state-of-the-art in most cases, convincingly showing its practicality and generality. The source code is available at https://github.com/ShiqiYu/OpenGait.

Motivation & Objective

• Question the effectiveness of shallow gait models on real-world data.
• Investigate the benefits of explicit temporal modeling for gait recognition.
• Evaluate CNN-based DeepGaitV2 versus Transformer-based SwinGait on constrained and real-world datasets.
• Address challenges unique to gait silhouettes for transformer Architectures.

Proposed method

• Proposes DeepGaitV2-3D/DeepGaitV2-2D backbones with 2D/3D residual units and pseudo-3D blocks to study depth effects.
• Introduces SwinGait-2D/3D transformers with convolutional front-ends to mitigate dumb-patch issues in silhouette tokens.
• Applies a local-window Swin Transformer with 2D/3D variants and a linear embedding to create tokens from transformed feature maps.
• Performs ablations varying depth (B) and width (C), and compares set-based versus sequence-based temporal modeling.
• Uses triplet loss with margin and standard data augmentation; evaluates on six gait datasets following official protocols.

Experimental results

Research questions

• RQ1 Can deep gait models (CNN- and Transformer-based) outperform shallow models on real-world gait data?
• RQ2 Does explicit temporal modeling improve gait recognition when appearance is unreliable?
• RQ3 Do transformer-based architectures (SwinGait) surpass CNN-based models on outdoor gait benchmarks?
• RQ4 How to mitigate non-informative (dumb) patches in gait silhouettes when using transformers?
• RQ5 What are the trade-offs between accuracy and efficiency for DeepGaitV2 versus SwinGait on constrained and real-world datasets?

Key findings

• DeepGaitV2-3D and DeepGaitV2-P3D achieve strong results on real-world datasets Gait3D and GREW, with significant improvements over prior methods.
• SwinGait-3D generally surpasses the corresponding CNN-based DeepGaitV2 models on outdoor datasets, offering a favorable speed–accuracy trade-off.
• Explicit temporal modeling (sequence-based) yields clear gains over set-based methods, as shown by performance differences when frame order is preserved.
• A practical solution to the dumb-patch problem in transformers is to use convolutional blocks early (Conv0 and early stages) before transformer layers, enabling effective tokenization of gait silhouettes.
• DeepGaitV2-P3D provides competitive or superior performance with far fewer parameters and FLOPs than 3D CNN counterparts on key datasets.