[Paper Review] Lip Movements Generation at a Glance

Lele Chen, Zhiheng Li+3 more|arXiv (Cornell University)|Mar 28, 2018

Speech and Audio Processing26 references3 citations

TL;DR

This paper proposes a novel end-to-end deep learning framework for generating realistic, identity-preserving lip movements from a single target identity image and arbitrary speech audio. By fusing audio and image embeddings and introducing a novel audio-visual correlation loss, the model generates 16-frame lip movement sequences with high temporal consistency, photo-realism, and identity fidelity, outperforming state-of-the-art methods on GRID, LDC, and LRW datasets in both quantitative and qualitative evaluations.

ABSTRACT

Cross-modality generation is an emerging topic that aims to synthesize data in one modality based on information in a different modality. In this paper, we consider a task of such: given an arbitrary audio speech and one lip image of arbitrary target identity, generate synthesized lip movements of the target identity saying the speech. To perform well in this task, it inevitably requires a model to not only consider the retention of target identity, photo-realistic of synthesized images, consistency and smoothness of lip images in a sequence, but more importantly, learn the correlations between audio speech and lip movements. To solve the collective problems, we explore the best modeling of the audio-visual correlations in building and training a lip-movement generator network. Specifically, we devise a method to fuse audio and image embeddings to generate multiple lip images at once and propose a novel correlation loss to synchronize lip changes and speech changes. Our final model utilizes a combination of four losses for a comprehensive consideration in generating lip movements; it is trained in an end-to-end fashion and is robust to lip shapes, view angles and different facial characteristics. Thoughtful experiments on three datasets ranging from lab-recorded to lips in-the-wild show that our model significantly outperforms other state-of-the-art methods extended to this task.

Motivation & Objective

• To generate high-fidelity lip movements for a target identity using only one reference image and arbitrary speech audio, without requiring video frames of the target in training.
• To model the complex temporal correlation between audio speech and lip movements across different identities.
• To ensure generated lip sequences are temporally consistent, photo-realistic, and preserve identity-specific characteristics such as beard or lip shape.
• To develop a training framework robust to variations in view angle, facial features, and lip geometry.
• To outperform existing methods in both quantitative metrics and qualitative realism on diverse datasets, including in-the-wild settings.

Proposed method

• The model uses a generator network that fuses time-series audio embeddings with a single-image identity embedding to generate 16 consecutive lip frames in a single forward pass.
• A novel audio-visual correlation loss is introduced to synchronize lip motion changes with speech signal changes, improving temporal alignment.
• The training employs a four-loss objective: reconstruction loss (pixel-level), feature-space loss (perceptual similarity), three-stream adversarial loss (image and motion quality), and the proposed audio-visual correlation loss.
• The discriminator is designed with three streams to jointly assess image quality, motion smoothness, and audio-visual alignment.
• The entire model is trained in an end-to-end fashion, enabling joint optimization of identity retention, visual quality, and audio-visual synchronization.
• The method is robust to variations in facial characteristics (e.g., beard), lip shapes, and viewing angles due to the disentangled embedding and multi-loss training.

Experimental results

Research questions

• RQ1 Can a single-reference-image-based lip movement generation model achieve high identity fidelity and temporal consistency without relying on video frames of the target in training?
• RQ2 How can audio-visual correlation be effectively modeled and enforced during end-to-end training to improve lip-speech synchronization?
• RQ3 To what extent can a model trained on limited data generalize to unseen identities and in-the-wild conditions?
• RQ4 What combination of losses best balances photo-realism, motion smoothness, and audio-visual alignment in lip movement generation?
• RQ5 How does the model perform under challenging conditions such as low-resolution input or unseen facial features?

Key findings

• The proposed model significantly outperforms state-of-the-art methods on all three datasets—GRID, LDC, and LRW—across multiple metrics, including LMD, PSNR, SSIM, and CPBD.
• The model achieves the best performance in landmark distance (LMD) and PSNR, indicating superior alignment and pixel-level accuracy compared to methods using only reconstruction loss or standard GANs.
• The CPBD metric shows that the model generates sharper images than the ground truth on the LRW dataset, likely due to the model's ability to enhance low-resolution inputs.
• Qualitative results demonstrate robust identity preservation, including distinguishing features like beards and facial asymmetries, even when the target identity is unseen during training.
• The model trained on the larger LRW dataset generalizes better than the one trained on GRID, particularly in preserving identity information for unseen individuals.
• Despite strong performance, the model occasionally fails to preserve skin tone accurately, likely due to data imbalance in the LRW dataset.