[Paper Review] BosphorusSign22k Sign Language Recognition Dataset

Oğulcan Özdemir, Ahmet Alp Kındıroğlu+2 more|arXiv (Cornell University)|Apr 2, 2020

Hand Gesture Recognition Systems48 references33 citations

TL;DR

Introduces the BosphorusSign22k large-scale Turkish Sign Language dataset, adds OpenPose and Kinect v2 modalities, defines an evaluation protocol, and provides baseline results using IDT and 3D ResNets (MC3).

ABSTRACT

Sign Language Recognition is a challenging research domain. It has recently seen several advancements with the increased availability of data. In this paper, we introduce the BosphorusSign22k, a publicly available large scale sign language dataset aimed at computer vision, video recognition and deep learning research communities. The primary objective of this dataset is to serve as a new benchmark in Turkish Sign Language Recognition for its vast lexicon, the high number of repetitions by native signers, high recording quality, and the unique syntactic properties of the signs it encompasses. We also provide state-of-the-art human pose estimates to encourage other tasks such as Sign Language Production. We survey other publicly available datasets and expand on how BosphorusSign22k can contribute to future research that is being made possible through the widespread availability of similar Sign Language resources. We have conducted extensive experiments and present baseline results to underpin future research on our dataset.

Motivation & Objective

• Provide a signer-independent isolated SLR benchmark for Turkish Sign Language.
• Clean and extend the BosphorusSign dataset with richer modalities (OpenPose, Kinect v2) and a workable labeling scheme.
• Define an evaluation protocol and establish baseline performance on the new dataset.
• Enable future research by offering baseline results with both hand-crafted and deep learning methods.

Proposed method

• Merged and cleaned the BosphorusSign dataset to remove erroneous recordings and collapsed signs with similar manual features.
• Provided RGB, depth, OpenPose joints (body, face, hands) and Kinect v2 skeleton data for all videos.
• Established a signer-independent evaluation protocol with a train/test split (one signer for testing, others for training).
• Benchmarked baselines using Improved Dense Trajectories (IDT) with Fisher Vector encoding and a 3D ResNet MC3 model with mixed 2D-3D convolutions.
• Compared training from scratch versus fine-tuning pre-trained networks on Kinetics-400, and evaluated various block finetuning strategies.
• Reported baseline results showing IDT achieving strong performance and analyzed failure modes to guide future improvements.

Experimental results

Research questions

• RQ1 What is the performance of state-of-the-art handcrafted video features (IDT) on BosphorusSign22k for Turkish Sign Language isolated recognition?
• RQ2 How do modern deep learning architectures (MC3 3D ResNets) perform on signer-independent Turkish sign gloss recognition compared to IDT?
• RQ3 What impact do data modalities (RGB, depth, OpenPose joints, Kinect skeleton) have on recognition performance?
• RQ4 Can a signer-independent baseline be established with a clear evaluation protocol for future benchmarking?
• RQ5 What are the main error patterns and challenges in distinguishing visually similar Turkish sign glosses?

Key findings

• IDT with HOF and MBH features achieves top-1 accuracy of 86.63% on individual motion components, and 88.53% when fusing HOG, HOF, and MBH.
• 3D ResNet MC3 baseline reaches 78.85% Top-1 accuracy when fine-tuning the last three blocks (and 94.76% Top-5).
• Training MC3 from scratch outperforms fine-tuning a pre-trained Kinetics-400 model for this task.
• Fusion of appearance (HOG) with motion features improves performance across baselines, while adding trajectory information can slightly decrease accuracy.
• The dataset provides 744 sign glosses across Health, Finance, and Common signs, with 22,542 videos from 6 signers (~19 hours).
• The authors identify that fine-grained hand shapes and longer-range temporal modeling remain key bottlenecks for signer-independent SLR.