[Paper Review] InstructIR: High-Quality Image Restoration Following Human Instructions

Marcos V. Conde, Gregor Geigle+1 more|arXiv (Cornell University)|Jan 29, 2024

Advanced Image Processing Techniques4 citations

TL;DR

InstructIR introduces the first text-guided image restoration model that uses natural language instructions to guide high-quality image restoration across multiple degradation types, including denoising, deraining, deblurring, dehazing, and low-light enhancement. By leveraging a frozen text encoder and a NAFNet-based image restoration backbone, InstructIR achieves state-of-the-art performance, improving by +1dB over prior all-in-one methods on key benchmarks.

ABSTRACT

Image restoration is a fundamental problem that involves recovering a high-quality clean image from its degraded observation. All-In-One image restoration models can effectively restore images from various types and levels of degradation using degradation-specific information as prompts to guide the restoration model. In this work, we present the first approach that uses human-written instructions to guide the image restoration model. Given natural language prompts, our model can recover high-quality images from their degraded counterparts, considering multiple degradation types. Our method, InstructIR, achieves state-of-the-art results on several restoration tasks including image denoising, deraining, deblurring, dehazing, and (low-light) image enhancement. InstructIR improves +1dB over previous all-in-one restoration methods. Moreover, our dataset and results represent a novel benchmark for new research on text-guided image restoration and enhancement. Our code, datasets and models are available at: https://github.com/mv-lab/InstructIR

Motivation & Objective

• To address the challenge of generalizing image restoration across diverse degradation types using natural language guidance.
• To enable a single all-in-one model to restore images based on user-provided textual instructions rather than predefined degradation embeddings.
• To establish a new benchmark for text-guided image restoration and enhancement using real human-written prompts.
• To improve generalization and performance in real-world scenarios by incorporating human intent through natural language.
• To demonstrate that text-based instruction following can outperform traditional prompt embedding or classification-based approaches in blind image restoration.

Proposed method

• The model uses a frozen text encoder (BGE-micro-v2) to embed human-written instructions into contextual embeddings.
• A NAFNet-based image restoration backbone processes the degraded image and the text embedding in a cross-attention mechanism to condition restoration on the instruction.
• The model is trained with a combined loss: L1 reconstruction loss between the restored and ground-truth image, and a cross-entropy loss for intent classification from the prompt.
• Text encoders are kept frozen during training to preserve generalization and reduce computational cost.
• The method supports arbitrary natural language prompts, enabling flexible and user-intent-driven image restoration.
• The model is evaluated on standard benchmarks for denoising, deraining, deblurring, dehazing, and low-light enhancement.

Experimental results

Research questions

• RQ1 Can natural language instructions effectively guide image restoration models across multiple degradation types?
• RQ2 Does using human-written instructions improve restoration performance compared to learned prompt embeddings or degradation classification?
• RQ3 Can a single all-in-one model generalize to diverse, real-world degradation scenarios using only textual guidance?
• RQ4 How does the choice of text encoder affect performance in text-guided image restoration?
• RQ5 Can the model achieve state-of-the-art results while maintaining generalization to unseen degradation types and instructions?

Key findings

• InstructIR achieves state-of-the-art performance across five image restoration tasks, improving by +1dB over previous all-in-one methods.
• The model generalizes well to real-world degradation scenarios, producing high-quality results even with ambiguous or non-technical instructions.
• Ablation studies show no significant performance difference between different text encoders, with BGE-micro-v2 achieving optimal trade-off between size and performance.
• The intent classification loss converges quickly, indicating that the model effectively learns to associate instructions with the correct restoration task.
• Qualitative comparisons show that InstructIR outperforms baselines like AirNet and PromptIR in visual quality and detail recovery across all tasks.
• The model demonstrates strong zero-shot generalization, restoring images based on diverse, human-written prompts without fine-tuning.