[Paper Review] Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Dongxu Li, Cristian Rodriguez Opazo+2 more|arXiv (Cornell University)|Oct 24, 2019

Hand Gesture Recognition SystemsComputer Science73 references53 citations

TL;DR

The paper introduces the large-scale Word-Level ASL (WLASL) dataset with over 21k videos across 2k glosses and compares appearance-based and pose-based baselines, proposing Pose-TGCN to jointly model spatial and temporal pose dynamics.

ABSTRACT

Vision-based sign language recognition aims at helping deaf people to communicate with others. However, most existing sign language datasets are limited to a small number of words. Due to the limited vocabulary size, models learned from those datasets cannot be applied in practice. In this paper, we introduce a new large-scale Word-Level American Sign Language (WLASL) video dataset, containing more than 2000 words performed by over 100 signers. This dataset will be made publicly available to the research community. To our knowledge, it is by far the largest public ASL dataset to facilitate word-level sign recognition research. Based on this new large-scale dataset, we are able to experiment with several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios. Specifically we implement and compare two different models,i.e., (i) holistic visual appearance-based approach, and (ii) 2D human pose based approach. Both models are valuable baselines that will benefit the community for method benchmarking. Moreover, we also propose a novel pose-based temporal graph convolution networks (Pose-TGCN) that models spatial and temporal dependencies in human pose trajectories simultaneously, which has further boosted the performance of the pose-based method. Our results show that pose-based and appearance-based models achieve comparable performances up to 66% at top-10 accuracy on 2,000 words/glosses, demonstrating the validity and challenges of our dataset. Our dataset and baseline deep models are available at \url{https://dxli94.github.io/WLASL/}.

Motivation & Objective

• Motivate and enable scalable word-level ASL recognition with a large, signer-diverse dataset collected from internet sources.
• Provide publicly available baselines for appearance-based and pose-based sign recognition to benchmark future work.
• Investigate the effectiveness of pose-based temporal graph networks (Pose-TGCN) versus appearance-based methods on large vocabularies.

Proposed method

• Construct a large-scale, monocular RGB Word-Level ASL dataset (WLASL) with 21,083 videos, 119 signers, and 3,126 glosses; ensure signer diversity and dialect annotations.
• Develop appearance-based baselines: 2D CNN (VGG16) + GRU, and 3D CNN (I3D) fine-tuned on Kinetics-derived features.
• Develop pose-based baselines: Pose-GRU using 55 2D keypoints with GRU; Pose-TGCN using temporal graph convolutions over full-body keypoint trajectories.
• Propose a Temporal Graph Convolution Network (TGCN) where the human body is modeled as a fully-connected graph with learnable adjacency matrices, stacked with residual blocks, and perform average pooling over time for classification.
• Standard training protocol: resize to 256 bounding-box diagonal; random 50-frame clips for training; Adam optimizer; 200 epochs; train/val/test split per gloss in 4:1:1 ratio.

Experimental results

Research questions

• RQ1 Can a large-scale, signer-diverse word-level ASL dataset enable robust learning for thousands of glosses?
• RQ2 How do appearance-based and pose-based approaches compare on large vocabulary word-level sign recognition?
• RQ3 Does a pose-based temporal graph approach (Pose-TGCN) outperform standard pose and appearance baselines in sign language recognition?
• RQ4 What is the impact of vocabulary size and sample counts on model performance in word-level SLR?

Key findings

• WLASL contains 21,083 videos for 2,000 glosses collected from 119 signers; the dataset is publicly available.
• Pose-TGCN achieves competitive top-10 performance with appearance-based models on large vocabularies (up to 62.24% top-10 on WLASL2000, comparable to I3D in some settings).
• I3D generally outs performs VGG-GRU, and Pose-TGCN improves over Pose-GRU, demonstrating the benefit of jointly modeling spatial and temporal pose information.
• On small vocabulary subsets, both pose-based and appearance-based methods perform better, but performance saturates as vocabulary size increases; larger vocabularies introduce more ambiguity and require more data or advanced learning strategies.