[Paper Review] SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning

Hao Chen, Jiaze Wang+7 more|arXiv (Cornell University)|Jan 22, 2024

Hand Gesture Recognition Systems4 citations

TL;DR

SignVTCL proposes a multi-modal continuous sign language recognition framework that enhances visual representations using video, keypoint, and optical flow data, combined with visual-textual contrastive learning at both gloss and sentence levels. It achieves state-of-the-art performance on Phoenix-2014, Phoenix-2014T, and CSL-Daily, reducing word error rates to 17.3% (dev) and 17.6% (test).

ABSTRACT

Sign language recognition (SLR) plays a vital role in facilitating communication for the hearing-impaired community. SLR is a weakly supervised task where entire videos are annotated with glosses, making it challenging to identify the corresponding gloss within a video segment. Recent studies indicate that the main bottleneck in SLR is the insufficient training caused by the limited availability of large-scale datasets. To address this challenge, we present SignVTCL, a multi-modal continuous sign language recognition framework enhanced by visual-textual contrastive learning, which leverages the full potential of multi-modal data and the generalization ability of language model. SignVTCL integrates multi-modal data (video, keypoints, and optical flow) simultaneously to train a unified visual backbone, thereby yielding more robust visual representations. Furthermore, SignVTCL contains a visual-textual alignment approach incorporating gloss-level and sentence-level alignment to ensure precise correspondence between visual features and glosses at the level of individual glosses and sentence. Experimental results conducted on three datasets, Phoenix-2014, Phoenix-2014T, and CSL-Daily, demonstrate that SignVTCL achieves state-of-the-art results compared with previous methods.

Motivation & Objective

• Address the challenge of limited large-scale annotated sign language datasets in continuous sign language recognition (CSLR).
• Improve visual representation learning by fusing video, keypoint, and optical flow modalities in a unified backbone.
• Leverage natural language supervision through gloss-level and sentence-level visual-textual alignment to enhance model generalization.
• Overcome the weakly supervised nature of SLR, where only full video glosses are annotated, by enabling fine-grained alignment between visual segments and textual glosses.
• Achieve state-of-the-art performance across multiple benchmarks through multi-modal and contrastive learning integration.

Proposed method

• Integrate three modalities—video, 2D pose keypoints, and optical flow—into a shared visual backbone for enhanced dynamic motion understanding.
• Employ a sign pyramid network (SPN) to generate auxiliary CTC losses at intermediate layers, improving feature learning across temporal scales.
• Implement a gloss-level alignment loss using dynamic time warping (DTW) to match visual features with individual glosses.
• Introduce a sentence-level alignment loss to enforce holistic semantic consistency between visual sequences and full sentence embeddings.
• Apply visual-textual contrastive learning by optimizing cross-modality similarity between visual and textual embeddings at both gloss and sentence levels.
• Use a multi-head decoder with CTC loss and auxiliary SPN supervision to train the model end-to-end on weakly annotated sign video data.

Experimental results

Research questions

• RQ1 Can joint learning from video, keypoint, and optical flow modalities significantly improve visual representation quality in sign language recognition?
• RQ2 How effective is visual-textual contrastive learning at the gloss level in aligning individual sign gestures with their corresponding textual glosses?
• RQ3 To what extent does sentence-level visual-textual alignment enhance contextual understanding and recognition accuracy in continuous sign language recognition?
• RQ4 Can the integration of multi-modal data and contrastive learning mitigate performance degradation due to limited large-scale annotated datasets in SLR?
• RQ5 What is the relative contribution of each modality (video, keypoints, optical flow) and training loss component to the final recognition performance?

Key findings

• SignVTCL achieves a word error rate (WER) of 17.3% on the development set and 17.6% on the test set of the Phoenix-2014 dataset, outperforming previous methods.
• The fusion of video, keypoint, and optical flow encoders reduces WER by 1.1% on the development set and 1.0% on the test set compared to using any single modality.
• The inclusion of the gloss-level alignment loss reduces WER from 18.3% to 17.5% on the development set, demonstrating its effectiveness in fine-grained alignment.
• Sentence-level alignment contributes to a 17.7% WER on the development set, indicating improved contextual modeling.
• The combination of SPN auxiliary loss and all alignment losses reduces WER to 17.3% (dev) and 17.6% (test), confirming the synergy of all components.
• Visualization of the gloss-level similarity matrix shows high concentration along the diagonal, confirming effective alignment between visual and textual representations.