[Paper Review] SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editing

Zijie Wu, Chaohui Yu+3 more|arXiv (Cornell University)|Oct 12, 2023

Video Analysis and Summarization4 citations

TL;DR

SingleInsert proposes a two-stage method for inserting new concepts from a single image into text-to-image diffusion models with high visual fidelity and flexible editing. By using foreground-aware inversion, background disentanglement, and a semantic loss to prevent language drift, it enables accurate concept editing, novel view synthesis, and multi-concept composition without joint training.

ABSTRACT

Recent progress in text-to-image (T2I) models enables high-quality image generation with flexible textual control. To utilize the abundant visual priors in the off-the-shelf T2I models, a series of methods try to invert an image to proper embedding that aligns with the semantic space of the T2I model. However, these image-to-text (I2T) inversion methods typically need multiple source images containing the same concept or struggle with the imbalance between editing flexibility and visual fidelity. In this work, we point out that the critical problem lies in the foreground-background entanglement when learning an intended concept, and propose a simple and effective baseline for single-image I2T inversion, named SingleInsert. SingleInsert adopts a two-stage scheme. In the first stage, we regulate the learned embedding to concentrate on the foreground area without being associated with the irrelevant background. In the second stage, we finetune the T2I model for better visual resemblance and devise a semantic loss to prevent the language drift problem. With the proposed techniques, SingleInsert excels in single concept generation with high visual fidelity while allowing flexible editing. Additionally, SingleInsert can perform single-image novel view synthesis and multiple concepts composition without requiring joint training. To facilitate evaluation, we design an editing prompt list and introduce a metric named Editing Success Rate (ESR) for quantitative assessment of editing flexibility. Our project page is: https://jarrentwu1031.github.io/SingleInsert-web/

Motivation & Objective

• Address the challenge of overfitting and poor editability when inserting a new concept from a single image into pre-trained text-to-image models.
• Overcome foreground-background entanglement in embedding learning that degrades editing flexibility and visual fidelity.
• Prevent language drift during fine-tuning, where class embeddings shift toward the source image, reducing generalizability.
• Enable novel view synthesis and multi-concept composition from independently learned embeddings without joint training.
• Provide a quantitative evaluation framework using an editing prompt list and a new metric, Editing Success Rate (ESR), for assessing editing flexibility.

Proposed method

• Use an image encoder to invert a single source image into a text embedding, optimizing only within the foreground mask obtained via pre-trained models (GroundingDINO and SAM).
• Introduce a foreground loss that restricts optimization to the foreground region, ensuring the embedding captures only the intended concept.
• Apply a background loss to minimize the influence of the learned embedding on the background denoising process, disentangling concept from background.
• Implement a semantic loss during fine-tuning to maintain the original class embedding’s influence on denoising, preventing language drift.
• Adopt a two-stage training scheme: first, invert the image using foreground and background losses; second, fine-tune the T2I model with the semantic loss to improve visual fidelity.
• Leverage pre-trained visual priors in the T2I model to generate plausible novel viewpoints and compose multiple concepts without retraining.

Experimental results

Research questions

• RQ1 How can a single-image concept insertion method achieve high visual fidelity while maintaining flexible editing capabilities?
• RQ2 What causes the entanglement of foreground and background in embedding learning, and how can it be mitigated?
• RQ3 How can language drift during fine-tuning be prevented to preserve class prior knowledge and editing flexibility?
• RQ4 Can a single learned concept be used for novel view synthesis and multi-concept composition without joint training?
• RQ5 What quantitative metrics can effectively assess the editing flexibility of concept insertion methods?

Key findings

• SingleInsert achieves high visual fidelity and strong editing flexibility by disentangling the learned concept from the background using a dedicated background loss.
• The semantic loss effectively prevents language drift, maintaining the original class embedding’s influence and preserving editability during fine-tuning.
• With the proposed method, novel view synthesis from a single image is possible, generating plausible novel viewpoints using the T2I model’s internal priors.
• Multiple concepts (e.g., face, hair, clothing) can be composed together without joint training, demonstrating compositional generalization across independently learned embeddings.
• The proposed Editing Success Rate (ESR) metric, combined with a curated editing prompt list, enables reliable quantitative evaluation of editing flexibility.
• Ablation studies confirm that all three losses—foreground, background, and semantic—are necessary for achieving the optimal balance between fidelity and editability.