[Paper Review] Progress in animation of an EMA-controlled tongue model for acoustic-visual speech synthesis

Ingmar Steiner, Slim Ouni|arXiv (Cornell University)|Jan 19, 2012

Phonetics and Phonology Research16 references3 citations

TL;DR

This paper presents a skeletal animation technique for a 3D kinematic tongue model in acoustic-visual speech synthesis, using electromagnetic articulography (EMA) motion capture data and MRI-derived geometry. The method enables realistic, data-driven tongue animation by mapping EMA coil trajectories to a deformable mesh via an embedded B-skeleton rig, achieving plausible articulation without biomechanical modeling.

ABSTRACT

We present a technique for the animation of a 3D kinematic tongue model, one component of the talking head of an acoustic-visual (AV) speech synthesizer. The skeletal animation approach is adapted to make use of a deformable rig controlled by tongue motion capture data obtained with electromagnetic articulography (EMA), while the tongue surface is extracted from volumetric magnetic resonance imaging (MRI) data. Initial results are shown and future work outlined.

Motivation & Objective

• To enhance visual naturalness and intelligibility in acoustic-visual text-to-speech (AV-TTS) systems by integrating a realistic 3D tongue model.
• To overcome the lack of intra-oral articulatory data in existing AV-TTS systems, which currently omit the tongue and teeth.
• To develop a data-driven, non-biomechanical approach to tongue animation using EMA motion capture and MRI-derived geometry.
• To enable real-time, synchronized animation of the tongue in a talking head GUI using off-the-shelf 3D software.
• To prepare the foundation for direct integration of the tongue model with the TTS system via action-based animation control.

Proposed method

• The tongue model is constructed from high-resolution 3D volumetric MRI data of the vocal tract, providing anatomical geometry.
• A skeletal rig composed of B-bones is embedded into the tongue mesh, with vertex groups assigned to each bone for deformation control.
• EMA motion capture data from a Carstens AG500 system (200 Hz) tracks 3D position and orientation of up to 12 coils on the tongue, providing sparse but rich kinematic data.
• Inverse kinematics (IK) is applied to the skeletal rig to animate the mesh based on EMA coil trajectories, enabling realistic deformation.
• The model is registered to the EMA data in a neutral bind pose, with manual alignment of coil positions to mesh vertices.
• The animation pipeline is exported using keyframe-based Non-Linear Animation (NLA) actions in Blender, enabling portability to lightweight game engines for real-time rendering.

Experimental results

Research questions

• RQ1 Can EMA motion capture data be effectively used to animate a 3D tongue model without biomechanical constraints?
• RQ2 How can sparse EMA data be mapped to a dense 3D mesh to produce plausible tongue deformations?
• RQ3 Can a skeletal animation approach with flexible B-bone rigging achieve realistic tongue motion when combined with MRI geometry?
• RQ4 What are the main challenges in aligning EMA coil positions with MRI-derived tongue meshes across different speakers?
• RQ5 How can EMA data errors and outliers be mitigated to ensure stable and natural-looking animation?

Key findings

• The technique successfully generates plausible tongue animations using EMA data and MRI geometry, with initial results showing visual plausibility in repetitive [ta] articulation.
• The use of orientation data from EMA coils helps compensate for the sparsity of positional data, improving animation stability.
• Manual registration of the MRI mesh to EMA coil positions results in a suboptimal fit, indicating a need for improved automatic registration methods.
• The skeletal animation approach with B-bones and IK enables realistic deformation of the tongue mesh without requiring physical simulation.
• The animation pipeline is exportable via standard formats (e.g., COLLADA), enabling integration into real-time game engines for efficient GUI rendering.
• Future work will focus on automatic registration, EMA data cleaning, and direct integration with the TTS system using action-based control.