[Paper Review] CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment

Jiangbin Zheng, Yile Wang+6 more|arXiv (Cornell University)|Mar 10, 2023

Hand Gesture Recognition Systems4 citations

TL;DR

This paper proposes CVT-SLR, a novel single-cue sign language recognition framework that leverages a variational autoencoder (VAE) to implicitly align visual and textual modalities while integrating complete pretrained language knowledge. By combining a contrastive cross-modal alignment loss with a video-gloss adapter, CVT-SLR achieves state-of-the-art performance on PHOENIX-2014 and PHOENIX-2014T, outperforming both single-cue and multi-cue SOTA methods despite a simpler architecture.

ABSTRACT

Sign language recognition (SLR) is a weakly supervised task that annotates sign videos as textual glosses. Recent studies show that insufficient training caused by the lack of large-scale available sign datasets becomes the main bottleneck for SLR. Most SLR works thereby adopt pretrained visual modules and develop two mainstream solutions. The multi-stream architectures extend multi-cue visual features, yielding the current SOTA performances but requiring complex designs and might introduce potential noise. Alternatively, the advanced single-cue SLR frameworks using explicit cross-modal alignment between visual and textual modalities are simple and effective, potentially competitive with the multi-cue framework. In this work, we propose a novel contrastive visual-textual transformation for SLR, CVT-SLR, to fully explore the pretrained knowledge of both the visual and language modalities. Based on the single-cue cross-modal alignment framework, we propose a variational autoencoder (VAE) for pretrained contextual knowledge while introducing the complete pretrained language module. The VAE implicitly aligns visual and textual modalities while benefiting from pretrained contextual knowledge as the traditional contextual module. Meanwhile, a contrastive cross-modal alignment algorithm is designed to explicitly enhance the consistency constraints. Extensive experiments on public datasets (PHOENIX-2014 and PHOENIX-2014T) demonstrate that our proposed CVT-SLR consistently outperforms existing single-cue methods and even outperforms SOTA multi-cue methods.

Motivation & Objective

• To address the data scarcity bottleneck in sign language recognition (SLR) by fully leveraging pretrained visual and language models.
• To replace the conventional contextual module in single-cue SLR with a variational autoencoder (VAE) that enables implicit cross-modal alignment and preserves full pretrained language knowledge.
• To enhance explicit cross-modal consistency through a contrastive learning-based alignment algorithm that distinguishes positive and negative sample pairs.
• To design a video-gloss adapter that maximizes preservation of pretrained visual and language parameters during fine-tuning.
• To demonstrate that a single-cue framework with advanced cross-modal alignment can surpass complex multi-cue SOTA methods in SLR.

Proposed method

• Replace the standard contextual module in single-cue SLR with a VAE-based module that learns from a pseudo-translation task using pretrained language models.
• Use the VAE’s autoencoding structure to implicitly align visual and textual features by reconstructing the input modality, thus enforcing consistency between modalities.
• Introduce a contrastive cross-modal alignment loss that maximizes similarity between positive visual-textual feature pairs and minimizes it for negative pairs.
• Design a video-gloss adapter to fine-tune the pretrained visual and language encoders without updating their original parameters, preserving knowledge transfer.
• Optimize the model jointly with CTC loss for sequence generation and contrastive alignment loss for cross-modal consistency.
• Apply GradCAM to visualize saliency maps and analyze attention focus on hand and facial regions, validating alignment with sign language linguistics.

Experimental results

Research questions

• RQ1 Can a VAE-based module effectively replace the standard contextual module in single-cue SLR while enabling implicit cross-modal alignment?
• RQ2 Does integrating a complete pretrained language model via a VAE improve SLR performance compared to standard contextual modules?
• RQ3 Can a contrastive alignment loss enhance explicit cross-modal consistency and lead to better recognition accuracy?
• RQ4 Does the proposed single-cue framework with implicit and explicit alignment surpass multi-cue SOTA methods in sign language recognition?
• RQ5 How does the use of pretrained visual and language modules affect training convergence speed and final performance?

Key findings

• CVT-SLR achieves new state-of-the-art performance on both PHOENIX-2014 and PHOENIX-2014T, outperforming existing single-cue and multi-cue methods.
• The ablation study confirms that the contrastive alignment loss significantly improves performance, with optimal performance at a loss weight of 10.0.
• Models with pretrained visual and textual modules converge faster and achieve lower training loss, correlating with better WER scores.
• Saliency maps show that the model focuses on key sign language regions—hands and face—validating alignment with linguistic principles.
• The VAE-based module with full pretrained language knowledge improves performance over standard RNN/LSTM modules, demonstrating the benefit of prior linguistic knowledge.
• The video-gloss adapter enables effective fine-tuning while preserving the full capacity of pretrained visual and language encoders.