[Paper Review] Synthesising Sign Language from semantics, approaching "from the target and back"

Michael Filhol, Gilles Falquet|arXiv (Cornell University)|Jul 25, 2017

Hand Gesture Recognition Systems8 references3 citations

TL;DR

This paper proposes AZee, a target-and-back approach to sign language synthesis that builds grammars directly from observed sign language forms, enabling direct generation of articulated signs from semantic input. By modeling function-to-form mappings through parametrized production rules derived from corpus data, AZee produces expressions closely aligned with semantic representations, outperforming source-and-forward methods in form generation fidelity and linguistic adequacy.

ABSTRACT

We present a Sign Language modelling approach allowing to build grammars and create linguistic input for Sign synthesis through avatars. We comment on the type of grammar it allows to build, and observe a resemblance between the resulting expressions and traditional semantic representations. Comparing the ways in which the paradigms are designed, we name and contrast two essentially different strategies for building higher-level linguistic input: "source-and-forward" vs. "target-and-back". We conclude by favouring the latter, acknowledging the power of being able to automatically generate output from semantically relevant input straight into articulations of the target language.

Motivation & Objective

• To address the challenge of synthesizing sign language from high-level semantic input without relying on pre-existing syntactic or lexical categories.
• To develop a formalism that captures systematic form-function mappings in sign language through empirical observation of corpus data.
• To compare and contrast two design strategies—source-and-forward vs. target-and-back—for building linguistic input for sign synthesis.
• To demonstrate that a grammar built from target language forms enables more reliable and semantically faithful sign generation than abstract semantic paradigms.
• To explore the feasibility of using AZee as a foundation for human-friendly, meaning-based sign language authoring tools.

Proposed method

• The AZee framework uses a minimal set of linguistic assumptions: observable forms carry meaning, systematic form-function links exist, and language is compositional.
• It models sign language through parametrized production rules of the form <H, P_i, f(P_i)>, where H is a function header, P_i are parameters, and f(P_i) specifies articulatory forms.
• Rules are extracted from annotated sign language corpora by identifying invariant form patterns associated with specific semantic functions, such as 'side-info' or 'seq'.
• Functional trees are constructed from AZee expressions, representing the hierarchical structure of semantic and articulatory components.
• Semantic graphs are compared with AZee trees to assess alignment between formal semantics and sign language articulation.
• The approach is evaluated by analyzing the correspondence between AZee-generated expressions and standard semantic representations, highlighting both overlaps and limitations in logical formalism.

Experimental results

Research questions

• RQ1 Can a sign language grammar be effectively built by observing systematic form-function mappings in real sign language data?
• RQ2 How does a target-and-back design, derived from actual sign language forms, compare to source-and-forward approaches in terms of semantic fidelity and form generation capability?
• RQ3 To what extent do AZee-generated expressions align with standard semantic representations, such as those based on description logics or semantic graphs?
• RQ4 What are the limitations of existing logical formalisms in representing sign language-specific features like focus, sequentiality, or non-manual markers?
• RQ5 Can AZee serve as a foundation for higher-level authoring tools that translate natural language meaning into sign language expressions?

Key findings

• AZee grammars, built from observed sign language forms, produce expressions that closely resemble traditional semantic representations, indicating strong alignment between semantic function and articulatory form.
• The target-and-back design ensures that every semantic function in the input corresponds to a directly implementable articulation rule, enabling trivial form generation from semantic input.
• Functions such as 'seq' and 'side-info'—which differ only in focus—cannot be trivially represented in standard logical formalisms, highlighting a key limitation of logic-based models.
• The approach avoids speculative linguistic categories by grounding all rules in empirical data, resulting in a grammar that is both minimal and linguistically adequate.
• Despite its strengths, AZee input requires significant abstraction, suggesting a need for intermediate layers to support human-friendly meaning representation.
• The method successfully captures complex relations such as agent, patient, and context through nested function structures, demonstrating robust compositional power.