[Paper Review] TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation

Dongxu Li, Chenchen Xu+5 more|arXiv (Cornell University)|Oct 12, 2020

Hand Gesture Recognition SystemsComputer Science28 references73 citations

TL;DR

TSPNet learns sign-language video representations from multi-scale segments using a temporal semantic pyramid with inter- and intra-scale attentions, improving sign-language translation without gloss annotations.

ABSTRACT

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of signvideos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96)on the largest commonly-used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.

Motivation & Objective

• Motivate reducing reliance on expensive gloss annotations in SLT by exploiting temporal structure in sign videos.
• Develop a multi-scale segment representation to capture both short- and long-range temporal semantics.
• Propose hierarchical feature learning with inter-scale attention for local semantic consistency and intra-scale attention for non-local context.
• Enable joint learning of local and non-local video semantics to mitigate segmentation noise and ambiguity.

Proposed method

• Create multi-scale sign video segments using windowed widths (e.g., 8, 12, 16 frames) and a sliding stride.
• Extract segment features with a fine-tuned I3D backbone on WSLR datasets.
• Introduce Shared Positional Embedding to encode segment positions across scales.
• Enforce local semantic consistency via inter-scale attention over a pivot segment and its larger-scale neighbors.
• Resolve local ambiguity with intra-scale self-attention over enriched pivot features.
• Optionally, jointly learn local and non-local semantics by extending the surrounding neighborhood to include all pivots (extended surrounding neighborhood).
• Use a Transformer decoder to generate translations from the encoder outputs.

Experimental results

Research questions

• RQ1 Can multi-scale sign video segments improve SLT over frame-wise features?
• RQ2 Does inter-scale attention improve local semantic consistency across scales, and does intra-scale attention leverage non-local context to reduce segmentation ambiguity?
• RQ3 Does joint learning of local and non-local semantics further enhance translation quality compared to sequential attention?
• RQ4 How does TSPNet perform on the RPWT dataset relative to prior bootstrapping models without gloss annotations?

Key findings

• TSPNet-Joint achieves the best translation scores on RPWT, with ROUGE-L 34.96 and BLEU-4 13.41.
• Multi-scale (8,12,16) segments outperform single-scale approaches and yield higher BLEU-4 and ROUGE-L.
• Inter-scale attention improves local semantic consistency by aggregating multi-scale segments.
• Intra-scale self-attention enhances non-local sentence context to resolve local gesture ambiguities.
• Joint local and non-local learning (TSPNet-Joint) surpasses sequential aggregation (TSPNet-Sequential).
• Compared to Conv2d-RNN, TSPNet variants show substantial BLEU-4 (13.41 vs 9.58) and ROUGE-L (34.96 vs 31.80) gains.
• Training requires about two hours on a single NVIDIA V100 GPU for TSPNet-Joint (excluding feature extraction).