[Paper Review] Self-Supervised Video Transformers for Isolated Sign Language Recognition

Marcelo Sandoval-Castañeda, Yanhong Li+3 more|arXiv (Cornell University)|Sep 2, 2023

Hand Gesture Recognition SystemsComputer Science3 citations

TL;DR

This paper proposes self-supervised video transformers for isolated sign language recognition (ISLR), leveraging masked reconstruction pre-training (MaskFeat) and hierarchical vision transformers to achieve state-of-the-art performance. On the WLASL2000 dataset, MViTv2 with MaskFeat pre-training reaches 79.02% top-1 accuracy, outperforming pose-based and supervised models by 1.59%, while linear probing reveals that hierarchical architectures better capture phonological features like handshape and movement.

ABSTRACT

This paper presents an in-depth analysis of various self-supervision methods for isolated sign language recognition (ISLR). We consider four recently introduced transformer-based approaches to self-supervised learning from videos, and four pre-training data regimes, and study all the combinations on the WLASL2000 dataset. Our findings reveal that MaskFeat achieves performance superior to pose-based and supervised video models, with a top-1 accuracy of 79.02% on gloss-based WLASL2000. Furthermore, we analyze these models' ability to produce representations of ASL signs using linear probing on diverse phonological features. This study underscores the value of architecture and pre-training task choices in ISLR. Specifically, our results on WLASL2000 highlight the power of masked reconstruction pre-training, and our linear probing results demonstrate the importance of hierarchical vision transformers for sign language representation.

Motivation & Objective

• To evaluate the effectiveness of self-supervised video transformers for isolated sign language recognition (ISLR) in low-data regimes.
• To investigate how different pre-training tasks and architectures impact sign language representation learning.
• To analyze model representations using phonological features from Brentari’s prosodic model of ASL.
• To establish a new state of the art on the gloss-based WLASL2000 benchmark.

Proposed method

• Pre-train vision transformers using four self-supervision methods: VideoMAE (pixel reconstruction), MaskFeat (feature reconstruction), BEVT (BERT-style video pre-training), and SVT (DINO-based contrastive learning).
• Fine-tune the pre-trained models on the WLASL2000 dataset for gloss-based sign classification using a linear probe and end-to-end training.
• Use hierarchical vision transformer architectures (e.g., MViTv2, VideoSwin) to better model spatial and temporal hierarchies in sign videos.
• Evaluate representations via linear probing on 15 phonological features from Brentari’s prosodic model, including handshape, movement, symmetry, and location.
• Train on two pre-training data regimes: Kinetics400 (action videos) and WLASL2000 (ASL videos), or a combination of both.
• Analyze layer-wise probing accuracy to assess how fine-tuning improves linguistic feature encoding across model depths.

Figure 1 : Sample frames from isolated sign language video extracted from the WLASL2000 dataset [ 19 ] .

Experimental results

Research questions

• RQ1 Which self-supervision pre-training task yields the best performance for ISLR on WLASL2000?
• RQ2 How do different vision transformer architectures (e.g., plain ViT vs. hierarchical) affect representation quality for sign language?
• RQ3 To what extent can self-supervised models learn linguistically meaningful features such as handshape and movement in sign language?
• RQ4 Does fine-tuning on ISLR improve the model’s ability to represent phonological features, even when not explicitly trained on them?
• RQ5 How does video sampling strategy during pre-training affect the model’s ability to capture time-dependent features like sign motion?

Key findings

• MaskFeat pre-training with MViTv2 achieves 79.02% top-1 accuracy on WLASL2000, setting a new state of the art and surpassing the previous pose-based SOTA by 1.59%.
• Hierarchical vision transformers (e.g., BEVT, MaskFeat) consistently outperform plain ViTs in linear probing across all 15 phonological features, indicating better linguistic representation learning.
• Fine-tuning on ISLR significantly improves linear probing accuracy for motion-based features like Path Movement, especially in models pre-trained on sparse sampling strategies.
• Models pre-trained on continuous signing videos (e.g., Kinetics400) struggle with time-dependent features due to sparse frame sampling, but fine-tuning on short, single-sign videos restores performance.
• Layer-wise probing shows that fine-tuning causes linear probing accuracy to monotonically increase toward the final layers, indicating improved linguistic feature encoding.
• MaskFeat and BEVT achieve performance comparable to or better than I3D (a strong CNN baseline) on all phonological features, while VideoMAE and SVT fail to match I3D on most features.

Figure 2 : Sample frames extracted from ASL to English translation dataset OpenASL [ 32 ] .