[Paper Review] ChatFace: Chat-Guided Real Face Editing via Diffusion Latent Space Manipulation

Dongxu Yue, Qin Guo+4 more|arXiv (Cornell University)|May 24, 2023

Face recognition and analysis4 citations

TL;DR

ChatFace introduces a novel, interactive system for high-quality, zero-shot editing of real facial images by leveraging the semantic latent space of diffusion models, guided by large language models (LLMs) and a Stable Manipulation Strategy (SMS) that ensures precise, identity-preserving edits. It outperforms prior methods in both quantitative metrics and human evaluation across multiple facial attributes.

ABSTRACT

Editing real facial images is a crucial task in computer vision with significant demand in various real-world applications. While GAN-based methods have showed potential in manipulating images especially when combined with CLIP, these methods are limited in their ability to reconstruct real images due to challenging GAN inversion capability. Despite the successful image reconstruction achieved by diffusion-based methods, there are still challenges in effectively manipulating fine-gained facial attributes with textual instructions.To address these issues and facilitate convenient manipulation of real facial images, we propose a novel approach that conduct text-driven image editing in the semantic latent space of diffusion model. By aligning the temporal feature of the diffusion model with the semantic condition at generative process, we introduce a stable manipulation strategy, which perform precise zero-shot manipulation effectively. Furthermore, we develop an interactive system named ChatFace, which combines the zero-shot reasoning ability of large language models to perform efficient manipulations in diffusion semantic latent space. This system enables users to perform complex multi-attribute manipulations through dialogue, opening up new possibilities for interactive image editing. Extensive experiments confirmed that our approach outperforms previous methods and enables precise editing of real facial images, making it a promising candidate for real-world applications. Project page: https://dongxuyue.github.io/chatface/

Motivation & Objective

• To address the limitations of GAN-based methods in reconstructing and editing real facial images due to unstable GAN inversion.
• To overcome the constraints of pre-defined editing directions in diffusion-based methods by enabling arbitrary, text-driven facial attribute manipulation.
• To develop an interactive, user-friendly system that interprets natural language queries and performs complex, multi-attribute edits on real faces.
• To ensure stable, semantically coherent edits by aligning temporal features of the diffusion model with semantic conditions during generation.
• To demonstrate the feasibility and superiority of combining LLMs with diffusion model semantics for fine-grained, real-world face editing.

Proposed method

• Utilizes a pre-trained Diffusion Autoencoder (DAE) to invert real facial images into a semantic latent code, enabling high-fidelity reconstruction.
• Employs a learned mapping network to transform text prompts into target latent codes in the diffusion model's semantic space.
• Introduces a Stable Manipulation Strategy (SMS) that aligns the temporal features of the diffusion model with the semantic condition at each denoising step, ensuring stable and consistent edits.
• Leverages a large language model (LLM) to parse complex, natural language editing queries and extract user intent for dynamic activation of editing directions.
• Performs linear interpolation in the semantic latent space between the original and target codes to generate edited images with minimal identity drift.
• Uses CLIP-based metrics (S_dir, SC, ID) and human evaluation to optimize and validate editing stability and realism.

Experimental results

Research questions

• RQ1 Can LLM-guided text prompts enable precise, zero-shot editing of real facial images without relying on GAN inversion?
• RQ2 How can the semantic latent space of a diffusion model be manipulated stably to preserve identity while changing specific facial attributes?
• RQ3 Can a multimodal system combining LLMs and diffusion models achieve superior performance in complex, multi-attribute face editing compared to existing methods?
• RQ4 What is the impact of the Stable Manipulation Strategy (SMS) on preserving low-level semantic details and preventing over-editing or identity collapse?
• RQ5 How does the system perform in real-world, interactive editing scenarios with diverse user queries?

Key findings

• ChatFace achieves a directional CLIP similarity (S_dir) of 0.21 on CelebA-HQ, outperforming all baselines including StyleCLIP (0.13), Stylegan-NADA (0.16), and DiffusionCLIP (0.18).
• The method achieves 89.7% segmentation consistency (SC) and 0.84 face identity similarity (ID), indicating strong preservation of semantic structure and identity.
• In human evaluation, ChatFace received 90.3% for smile, 82.7% for curly hair, 90.4% for makeup, and 94.2% for glasses, significantly outperforming other methods in all attributes.
• Ablation studies confirm that SMS is essential: without it, edits show color overflow (e.g., pink lipstick spreading beyond lips) and poor preservation of low-level image details.
• The system demonstrates continuous editing capability, allowing users to perform sequential, complex edits through natural language dialogue.
• The integration of LLMs enables effective intent understanding and dynamic activation of editing directions, enhancing usability and flexibility.