[Paper Review] Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

Ben Saunders, Necati Cihan Camgöz+1 more|arXiv (Cornell University)|Nov 19, 2020

Hand Gesture Recognition Systems65 references36 citations

TL;DR

SignGAN translates spoken language into continuous, photo-realistic sign language videos by first generating skeleton poses via a Transformer+MDN, then rendering a photo-realistic signer with controllable multi-signer styles and a hand-focused keypoint loss to improve hand quality.

ABSTRACT

To be truly understandable and accepted by Deaf communities, an automatic Sign Language Production (SLP) system must generate a photo-realistic signer. Prior approaches based on graphical avatars have proven unpopular, whereas recent neural SLP works that produce skeleton pose sequences have been shown to be not understandable to Deaf viewers. In this paper, we propose SignGAN, the first SLP model to produce photo-realistic continuous sign language videos directly from spoken language. We employ a transformer architecture with a Mixture Density Network (MDN) formulation to handle the translation from spoken language to skeletal pose. A pose-conditioned human synthesis model is then introduced to generate a photo-realistic sign language video from the skeletal pose sequence. This allows the photo-realistic production of sign videos directly translated from written text. We further propose a novel keypoint-based loss function, which significantly improves the quality of synthesized hand images, operating in the keypoint space to avoid issues caused by motion blur. In addition, we introduce a method for controllable video generation, enabling training on large, diverse sign language datasets and providing the ability to control the signer appearance at inference. Using a dataset of eight different sign language interpreters extracted from broadcast footage, we show that SignGAN significantly outperforms all baseline methods for quantitative metrics and human perceptual studies.

Motivation & Objective

• Motivate the need for photo-realistic Sign Language Production (SLP) that is truly understandable to Deaf communities, beyond skeletal representations.
• Propose a joint pipeline that translates spoken language to continuous sign poses and then to high-quality signer videos.
• Address hand image quality and signer variability through novel losses and controllable generation.
• Enable training on diverse datasets by separating content (pose) and style (signer appearance).
• Evaluate against baselines on back-translation and perceptual metrics to demonstrate improvements.

Proposed method

• Use a Transformer-based continuous sign language production module with a Mixture Density Network (MDN) to model multi-modal pose distributions conditioned on spoken language.
• Convert produced sign poses into heat-map representations to condition the video synthesis network.
• Employ a pose-conditioned video-to-video synthesis module with a multi-scale GAN (generator G and discriminators D1–D3) to produce photo-realistic signer videos, with skip connections in a U-Net style.
• Introduce a hand keypoint loss using a pre-trained hand pose estimator to supervise hand regions in keypoint space, mitigating motion blur effects in hand synthesis.
• Implement controllable video generation by conditioning on a signer style image, enabling multi-signer outputs and appearance control at inference.
• Combine adversarial, feature-m matching, and perceptual losses, plus a temporal consistency loss, into a total objective L_Total for video synthesis.
• Train on separate datasets for SLP (PHOENIX14T) and signer video generation to leverage data quality differences, with pose normalization to align datasets.

Experimental results

Research questions

• RQ1 Can a single end-to-end pipeline translate spoken language into photo-realistic sign language video while preserving signer identity and appearance?
• RQ2 Does introducing an MDN-based multi-modal pose model improve continuous sign language production over deterministic approaches?
• RQ3 Does a pose-conditioned, controllable signer generation improve perceptual realism and signer consistency across multiple signers?
• RQ4 Does a hand keypoint-based loss improve hand quality and reduce blur compared with standard discriminators on hands?
• RQ5 How does SignGAN perform against state-of-the-art baselines in back-translation and perceptual studies?

Key findings

• SignGAN achieves state-of-the-art back-translation BLEU-4 on the PHOENIX14T dataset for Text to Pose translation (12.18 on test set).
• SignGAN outperforms baselines in pose-conditioned signer generation across full metrics (SSIM, Hand SSIM, Hand Pose distance, FID) on both single and multiple signer datasets.
• The hand keypoint loss significantly improves hand region quality and Hand SSIM compared with a standard hand discriminator.
• Controllable generation with multiple signer styles yields better SSIM and FID and improves perceptual preferences in user studies.
• MDN-based multi-modal pose modeling reduces regression-to-the-mean and yields more expressive sign pose sequences than deterministic approaches.
• Subjective perceptual study shows SignGAN preferred over baselines in body and hands across both single and multiple-signer setups (high preference percentages).