[Paper Review] Multi-Modal Zero-Shot Sign Language Recognition

Razieh Rastgoo, Kourosh Kiani+2 more|arXiv (Cornell University)|Sep 2, 2021

Hand Gesture Recognition Systems39 references4 citations

TL;DR

This paper proposes a multi-modal, hybrid-feature, embedding-based zero-shot sign language recognition model that fuses deep features from C3D and skeleton-based features (distances, angles, SVD) via an autoencoder and LSTM, mapping them to BERT-embedded class labels. The model achieves state-of-the-art performance on four large-scale datasets, demonstrating robustness through multi-modal fusion and effective handling of unseen sign classes without annotated visual examples.

ABSTRACT

Zero-Shot Learning (ZSL) has rapidly advanced in recent years. Towards overcoming the annotation bottleneck in the Sign Language Recognition (SLR), we explore the idea of Zero-Shot Sign Language Recognition (ZS-SLR) with no annotated visual examples, by leveraging their textual descriptions. In this way, we propose a multi-modal Zero-Shot Sign Language Recognition (ZS-SLR) model harnessing from the complementary capabilities of deep features fused with the skeleton-based ones. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction, respectively. To make a trade-off between the dimensionality of the skeletonbased and deep features, we use an Auto-Encoder (AE) on top of the Long Short Term Memory (LSTM) network. Finally, a semantic space is used to map the visual features to the lingual embedding of the class labels, achieved via the Bidirectional Encoder Representations from Transformers (BERT) model. Results on four large-scale datasets, RKS-PERSIANSIGN, First-Person, ASLVID, and isoGD, show the superiority of the proposed model compared to state-of-the-art alternatives in ZS-SLR.

Motivation & Objective

• To address the annotation bottleneck in sign language recognition by enabling zero-shot recognition without any annotated visual examples of unseen classes.
• To improve robustness and generalization in sign language recognition by leveraging complementary multi-modal inputs: RGB video, 3D skeleton, and textual descriptions.
• To develop a hybrid feature representation combining hand-crafted skeleton features (distances, angles, SVD) with deep features from a C3D network.
• To enhance feature dimensionality balance and representation learning using an autoencoder on top of an LSTM encoder for visual features.
• To map visual features into a shared semantic space with BERT-embedded class labels for zero-shot generalization.

Proposed method

• A Transformer-based model is configured for real-time, accurate hand detection to improve input quality for sign recognition.
• Deep features are extracted from RGB video using a pre-trained C3D 3D CNN model, while skeleton features (distances, joint angles, SVD of joint positions) are computed from 3D pose data.
• An autoencoder is applied to the LSTM-embedded deep features to reduce dimensionality and balance the feature space with skeleton features.
• The fused visual features (deep + skeleton) are projected into a shared semantic space aligned with BERT-embedded class labels using a contrastive learning objective.
• The model uses a multi-modal input pipeline integrating RGB video, 3D skeleton sequences, and textual descriptions of sign labels to enable zero-shot generalization.
• A two-stage evaluation protocol is used to validate performance on unseen sign classes using four large-scale datasets: RKS-PERSIANSIGN, First-Person, ASLVID, and isoGD.

Experimental results

Research questions

• RQ1 Can a multi-modal, hybrid-feature approach improve zero-shot sign language recognition performance by fusing deep and skeleton-based features?
• RQ2 How effective is the use of BERT-embedded textual descriptions to map visual features into a shared semantic space for zero-shot generalization?
• RQ3 Does the integration of a Transformer-based hand detection model improve the robustness and accuracy of sign recognition in low-resource settings?
• RQ4 To what extent does feature dimensionality balancing via an autoencoder enhance model performance and generalization in ZS-SLR?
• RQ5 How does the proposed model compare to state-of-the-art methods in terms of zero-shot accuracy and robustness across diverse sign language datasets?

Key findings

• The proposed model achieves state-of-the-art performance on four large-scale sign language datasets: RKS-PERSIANSIGN, First-Person, ASLVID, and isoGD, outperforming existing SOTA methods in zero-shot sign language recognition.
• The model demonstrates superior generalization by recognizing unseen sign classes using only textual descriptions, with no annotated visual examples during training.
• The fusion of deep features (C3D) and skeleton-based features (distances, angles, SVD) significantly improves recognition accuracy, especially in complex or ambiguous sign categories.
• The use of an autoencoder to balance the dimensionality of deep and skeleton features leads to more stable and effective feature representation learning.
• The model shows higher recognition accuracy on RKS-PERSIANSIGN (mean accuracy >0.7 per sign) compared to ASLVID, with challenging signs like 'Sister' vs 'Brother' and 'Alligator' vs 'Sailboat' being common sources of misclassification.
• False positives are reduced by leveraging multi-modal complementarity: when one modality fails, the others compensate, increasing robustness and reducing bias toward any single feature type.