[Paper Review] A Data Augmentation Approach for Sign-Language-To-Text Translation In-The-Wild

Xiujun Li, Xi Yin+10 more|arXiv (Cornell University)|Apr 13, 2020

Multimodal Machine Learning Applications48 references134 citations

TL;DR

This paper proposes Oscar, a vision-language pre-training method that uses object tags detected in images as semantic anchor points to improve cross-modal alignment. By integrating object tags with region features and text embeddings in a unified Transformer framework, Oscar achieves state-of-the-art performance on six vision-language tasks, outperforming prior methods by significant margins on benchmarks including VQA, image captioning, and NLVR2.

ABSTRACT

In this paper, we describe the current main approaches to sign language translation which use deep neural networks with videos as input and text as output. We highlight that, under our point of view, their main weakness is the lack of generalization in daily life contexts. Our goal is to build a state-of-the-art system for the automatic interpretation of sign language in unpredictable video framing conditions. Our main contribution is the shift from image features to landmark positions in order to diminish the size of the input data and facilitate the combination of data augmentation techniques for landmarks. We describe the set of hypotheses to build such a system and the list of experiments that will lead us to their verification.

Motivation & Objective

• Address the challenge of weakly-supervised cross-modal alignment in vision-language pre-training, where image regions and text tokens lack explicit grounding.
• Leverage the high accuracy of modern object detectors to identify salient, semantically meaningful objects as anchor points for alignment.
• Improve the efficiency and effectiveness of learning image-text semantic alignments by providing explicit, grounded supervision via object tags.
• Achieve state-of-the-art performance on diverse vision-language understanding and generation tasks through improved pre-training.
• Demonstrate that object tags are more effective as alignment anchors than as direct visual features, by analyzing their role in representation learning.

Proposed method

• Represent each image-text pair as a triple: word tokens, object tags (detected via object detector), and image region features.
• Use pre-trained word embeddings for both text and object tags, aligning them in a shared semantic space via dictionary lookup.
• Pre-train the model using two objectives: masked token prediction over words and tags, and contrastive loss between object tags and their corresponding image regions.
• Integrate object tags as anchor points to guide self-attention mechanisms in the Transformer encoder, improving alignment between visual regions and linguistic units.
• Leverage the linguistic semantic space (e.g., BERT embeddings) to enhance the representational quality of object tags, making them more discriminative than raw region features.
• Fine-tune the pre-trained Oscar model on downstream tasks including VQA, image captioning, and image-text retrieval, using standard fine-tuning protocols.

Experimental results

Research questions

• RQ1 Can object tags detected in images serve as effective anchor points to improve cross-modal alignment in vision-language pre-training?
• RQ2 How does using object tags as semantic anchors compare to relying solely on self-attention over raw region features and text tokens?
• RQ3 Does pre-training with object tags lead to improved performance on both vision-language understanding and generation tasks?
• RQ4 What is the impact of different object tag sources (e.g., Visual Genome vs. Open Images) on downstream performance?
• RQ5 Can object tags improve generalization in zero-shot or few-shot settings, such as novel object captioning?

Key findings

• Oscar achieves new state-of-the-art results on six established vision-language benchmarks, including VQA, image captioning, and NLVR2.
• On the VQA dev set, Oscar improves accuracy to 71.70, compared to 70.93 in the baseline without object tags, a relative gain of 1.1%.
• In image captioning, Oscar achieves a BLEU-4 score of 36.4, ROUGE-L of 30.3, and a CIDEr score of 123.4, outperforming the baseline by 1.9, 1.2, and 7.8 points respectively.
• The use of object tags as anchor points significantly improves performance over baselines that do not use them, even when using the same region features.
• Models pre-trained with Visual Genome (VG) object tags outperform those using Open Images (OI) tags, likely due to greater object diversity in the VG dataset.
• Ablation studies confirm that object tags are most effective when used as alignment anchors rather than as direct visual features, as they improve alignment without increasing feature redundancy.