[Paper Review] HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models

Hayk Manukyan, Andranik Sargsyan+4 more|arXiv (Cornell University)|Dec 21, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

HD-Painter is a training-free, high-resolution text-guided image inpainting method that improves prompt fidelity using two novel components: Prompt-Aware Introverted Attention (PAIntA), which enhances attention alignment with textual prompts, and Reweighting Attention Score Guidance (RASG), a post-hoc sampling strategy that prevents distributional shifts. It achieves a 61.4% generation accuracy on MSCOCO, significantly outperforming prior SOTA methods.

ABSTRACT

Recent progress in text-guided image inpainting, based on the unprecedented success of text-to-image diffusion models, has led to exceptionally realistic and visually plausible results. However, there is still significant potential for improvement in current text-to-image inpainting models, particularly in better aligning the inpainted area with user prompts and performing high-resolution inpainting. Therefore, we introduce HD-Painter, a training free approach that accurately follows prompts and coherently scales to high resolution image inpainting. To this end, we design the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention scores by prompt information resulting in better text aligned generations. To further improve the prompt coherence we introduce the Reweighting Attention Score Guidance (RASG) mechanism seamlessly integrating a post-hoc sampling strategy into the general form of DDIM to prevent out-of-distribution latent shifts. Moreover, HD-Painter allows extension to larger scales by introducing a specialized super-resolution technique customized for inpainting, enabling the completion of missing regions in images of up to 2K resolution. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively across multiple metrics and a user study. Code is publicly available at: https://github.com/Picsart-AI-Research/HD-Painter

Motivation & Objective

• To address prompt neglect in text-guided image inpainting, particularly background dominance and nearby object propagation.
• To enable high-resolution inpainting up to 2048×2048 without compromising prompt alignment or visual quality.
• To develop a training-free, plug-and-play solution that enhances existing diffusion models for better text-image alignment.
• To improve generation fidelity and user-perceived quality through a specialized super-resolution pipeline for inpainted regions.

Proposed method

• PAIntA enhances self-attention scores by integrating textual prompt embeddings, reducing influence from irrelevant image features and increasing focus on prompt-relevant content.
• RASG modifies the DDIM sampling process by reweighting attention scores during inference to guide generation toward prompt-aligned latents while avoiding out-of-distribution shifts.
• The method uses a time-iterative blending strategy with high-resolution diffusion models to maintain coherence during high-resolution generation.
• A specialized super-resolution technique is applied to seamlessly upscale inpainted regions, preserving details from known regions.
• Both PAIntA and RASG are plug-and-play, enabling integration into any pre-trained diffusion-based inpainting model without retraining.
• The approach leverages CLIP, MMDetection, and PickScore for comprehensive evaluation of prompt alignment and visual quality.

Experimental results

Research questions

• RQ1 Can a training-free method significantly improve prompt fidelity in text-guided image inpainting without fine-tuning?
• RQ2 How can attention mechanisms be modified to prioritize textual prompts over visual context in masked image regions?
• RQ3 Can post-hoc guidance be designed to avoid distributional shifts while enhancing prompt alignment in diffusion sampling?
• RQ4 What is the impact of specialized super-resolution on high-resolution inpainting quality and coherence?
• RQ5 How does the combination of PAIntA and RASG compare to existing SOTA methods in both quantitative metrics and user perception?

Key findings

• HD-Painter achieves a 61.4% generation accuracy on MSCOCO, a 9.5 percentage point improvement over the previous SOTA of 51.9%.
• The method improves CLIP score by over 1.5 points compared to all competitors, indicating stronger text-image alignment.
• PickScore analysis shows HD-Painter outperforms all baselines in overall quality and user preference, with a statistically significant advantage.
• User studies confirm that HD-Painter is significantly preferred over competitors in both prompt alignment and visual quality.
• Qualitative results demonstrate consistent success in generating target objects even when the prompt contradicts visual context, avoiding background or object dominance.
• The inpainting-specific super-resolution technique enables seamless high-resolution completion, with superior detail preservation compared to vanilla super-resolution.