[Paper Review] DreamIdentity: Improved Editability for Efficient Face-identity Preserved Image Generation

Zhuowei Chen, Shancheng Fang+5 more|arXiv (Cornell University)|Jul 1, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

DreamIdentity proposes an optimization-free, efficient method for face-identity preserved image generation by introducing an M² ID encoder that uses multi-scale, face-specific features and multi-word embeddings to improve identity representation. It further enhances editability through self-augmented editability learning using the T2I model to generate paired editing data, achieving high identity preservation and flexible text-guided editing without test-time optimization.

ABSTRACT

While large-scale pre-trained text-to-image models can synthesize diverse and high-quality human-centric images, an intractable problem is how to preserve the face identity for conditioned face images. Existing methods either require time-consuming optimization for each face-identity or learning an efficient encoder at the cost of harming the editability of models. In this work, we present an optimization-free method for each face identity, meanwhile keeping the editability for text-to-image models. Specifically, we propose a novel face-identity encoder to learn an accurate representation of human faces, which applies multi-scale face features followed by a multi-embedding projector to directly generate the pseudo words in the text embedding space. Besides, we propose self-augmented editability learning to enhance the editability of models, which is achieved by constructing paired generated face and edited face images using celebrity names, aiming at transferring mature ability of off-the-shelf text-to-image models in celebrity faces to unseen faces. Extensive experiments show that our methods can generate identity-preserved images under different scenes at a much faster speed.

Motivation & Objective

• Address the challenge of preserving face identity in text-to-image generation without requiring per-identity optimization.
• Overcome the trade-off between identity preservation and model editability in existing optimization-free methods.
• Improve identity representation by replacing generic encoders (e.g., CLIP) with a face-specific, multi-scale feature encoder.
• Align training and inference objectives to maintain editability by incorporating editing tasks into the training phase.
• Enable efficient, real-time generation of diverse, identity-consistent images across different scenes using only a single input face image.

Proposed method

• Propose an M² ID encoder based on Vision Transformer that processes multi-scale features from a face image and projects them into multiple word embeddings in the text embedding space.
• Use a multi-embedding projector to generate multiple pseudo-words that better capture identity-specific details than single embeddings.
• Introduce self-augmented editability learning by leveraging the pre-trained T2I model to generate paired images: original celebrity faces and edited versions (e.g., 'as a police officer') using natural language prompts.
• Train the M² ID encoder on a combined dataset of real face reconstructions (from FFHQ) and self-augmented editing pairs to improve both identity fidelity and editability.
• Maintain inference efficiency by avoiding any optimization per identity, enabling one forward pass per input face.
• Integrate the M² ID encoder with ControlNet for spatial control, enabling identity-preserving scene switching with gaze-guided layout control.

Experimental results

Research questions

• RQ1 Can a lightweight, optimization-free encoder achieve superior identity preservation compared to CLIP-based baselines?
• RQ2 Does self-augmented editability learning improve the model’s ability to follow diverse text prompts while maintaining identity?
• RQ3 How does multi-scale feature extraction and multi-word embedding projection affect identity representation accuracy?
• RQ4 What is the optimal number of word embeddings for balancing identity preservation and text alignment?
• RQ5 Can the model generalize to unseen identities and complex editing scenarios without fine-tuning?

Key findings

• The M² ID encoder improves face identity preservation from 0.266 (CLIP baseline) to 0.412 by leveraging multi-scale, face-aware features.
• Using two word embeddings achieves a 12% improvement in face similarity with only a 0.4% drop in text-alignment, while more embeddings degrade editability.
• Self-augmented editability learning significantly enhances editing capability: models trained without it fail to generate correct edits (e.g., 'as a police officer').
• Combining reconstruction data (FFHQ) with self-augmented editing data yields better face similarity and prompt adherence than using either alone.
• The method achieves high-quality, identity-preserving image generation in a single forward pass, with no test-time optimization required.
• Qualitative results show consistent identity retention across diverse scenes and editing prompts, including changes in attire, pose, and background.