[Paper Review] A Benchmark and Baseline for Language-Driven Image Editing

Jing Shi, Ning Xu+4 more|arXiv (Cornell University)|Oct 5, 2020

Multimodal Machine Learning Applications29 references5 citations

TL;DR

This paper introduces a new language-driven image editing (LDIE) task that supports both local and global edits using natural language requests. It proposes the first large-scale benchmark, GIER, with 30k annotated image-triplets (source image, request, target image), and a modular neural network baseline that interprets requests, predicts editing operations and masks, and applies them step-by-step with high interpretability and strong performance on real user data.

ABSTRACT

Language-driven image editing can significantly save the laborious image editing work and be friendly to the photography novice. However, most similar work can only deal with a specific image domain or can only do global retouching. To solve this new task, we first present a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations. Besides, we also propose a baseline method that fully utilizes the annotation to solve this problem. Our new method treats each editing operation as a sub-module and can automatically predict operation parameters. Not only performing well on challenging user data, but such an approach is also highly interpretable. We believe our work, including both the benchmark and the baseline, will advance the image editing area towards a more general and free-form level.

Motivation & Objective

• To address the challenge of making image editing accessible to non-experts by enabling natural language-based editing without manual operation selection or parameter tuning.
• To create a large-scale, real-world dataset supporting both local and global image editing with detailed annotations for operations and masks.
• To develop a modular, interpretable model that predicts editing operations, regions, and parameters from language requests, enabling human-in-the-loop refinement.
• To establish a strong baseline for future research in free-form, language-driven image editing.

Proposed method

• Proposes the Grounded Image Editing Request (GIER) dataset with 30k samples from real user requests on platforms like Reddit and Zhopped, including source images, natural language editing requests, and target images.
• Designs a modular neural network (OMN) that parses language requests into a sequence of editing operations, each with predicted regions and parameters.
• Introduces an operation-conditioned grounding model that extends MattNet to handle multiple regions and operation-specific context, improving localization accuracy.
• Uses triplet loss during training to align features across image, language, and operation modalities, enhancing joint representation learning.
• Employs a two-stage training process: first predicting operations and masks, then refining the output image using a GAN-based generator.
• Validates the model using ablation studies and human evaluation, demonstrating robustness on diverse, real-world user requests.

Experimental results

Research questions

• RQ1 Can a language-driven image editing system effectively interpret both detailed and vague natural language requests for both local and global image edits?
• RQ2 How does the inclusion of operation and mask annotations improve the performance and interpretability of image editing models?
• RQ3 To what extent can a modular, operation-specific network architecture outperform end-to-end GAN-based models in terms of perceptual quality and edit accuracy?
• RQ4 How do visual and language features contribute to operation prediction and grounding, and which modality is more critical for local vs. global edits?
• RQ5 Can intermediate supervision from predicted operations and masks improve the quality of final image generation and help validate dataset annotations?

Key findings

• The proposed modular network (OMN) achieves strong performance on real user data, with a user rating significantly higher than a GAN-based baseline (Pix2pixAug), indicating better perceptual quality and human-interaction potential.
• The model's upper bound performance (with ground-truth operations and masks) is much higher than the full model, indicating that improved operation and mask prediction could yield substantial gains.
• Ablation studies show that using both vision and language features (V+L) slightly underperforms using only language features (L) for operation selection, suggesting language context alone is highly informative for operation prediction.
• The triplet loss during training improves performance, confirming its effectiveness in aligning multimodal representations across image, language, and operation spaces.
• Randomizing the operation order during training slightly improves performance over fixed order, indicating that model generalization benefits from diverse operation sequencing.
• The model demonstrates strong local editing capabilities, correctly removing objects like text and people in complex scenes, outperforming Pix2pixAug in visual grounding and edit saliency.