[Paper Review] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

Jiazheng Xu, Xiao Liu+6 more|arXiv (Cornell University)|Apr 12, 2023

Multimodal Machine Learning ApplicationsComputer Science99 citations

TL;DR

ImageReward is a general-purpose human-preference reward model for text-to-image generation, paired with Reward Feedback Learning (ReFL) to directly tune diffusion models using human-preference feedback.

ABSTRACT

We present a comprehensive solution to learn and improve text-to-image models from human preference feedback. To begin with, we build ImageReward -- the first general-purpose text-to-image human preference reward model -- to effectively encode human preferences. Its training is based on our systematic annotation pipeline including rating and ranking, which collects 137k expert comparisons to date. In human evaluation, ImageReward outperforms existing scoring models and metrics, making it a promising automatic metric for evaluating text-to-image synthesis. On top of it, we propose Reward Feedback Learning (ReFL), a direct tuning algorithm to optimize diffusion models against a scorer. Both automatic and human evaluation support ReFL's advantages over compared methods. All code and datasets are provided at \url{https://github.com/THUDM/ImageReward}.

Motivation & Objective

• Address the misalignment between text-to-image generation and human preferences.
• Create a general-purpose human-preference reward model for text-to-image tasks.
• Provide a scalable evaluation metric aligned with human judgments.
• Propose a direct optimization method (ReFL) to tune diffusion models using the reward scorer.
• Demonstrate improvements over existing scoring metrics and tuning methods.

Proposed method

• Develop a systematic human-preference annotation pipeline (prompt selection, text-image rating, image ranking) and build a 137k-comparison dataset.
• Train a reward model (ImageReward) using a BLIP backbone with a ranking-based loss over prompt-image pairs.
• Evaluate ImageReward against CLIP, Aesthetic, and BLIP as scoring baselines, and demonstrate superior alignment with human preferences.
• Propose Reward Feedback Learning (ReFL): directly fine-tune diffusion models by backpropagating rewards from ImageReward at later denoising steps.
• Use a random late denoising step for feedback to stabilize training, and combine reward loss with pretraining loss for regularization.
• Experimentally validate ReFL on Stable Diffusion v1.4 and compare with other tuning approaches.

Experimental results

Research questions

• RQ1 Can a general-purpose reward model reliably capture human preferences for text-to-image outputs?
• RQ2 Does ImageReward align with human judgments better than existing scoring metrics (CLIP, Aesthetic, BLIP)?
• RQ3 Can ReFL directly improve diffusion models using the reward signal without relying on data augmentation or loss reweighting alone?
• RQ4 How does the annotation pipeline affect quality and agreement in human preference data?
• RQ5 Is ImageReward effective as an automatic evaluation metric for single-image or model-wide comparisons?

Key findings

• ImageReward outperforms CLIP, Aesthetic, and BLIP by 38.6%, 39.6%, and 31.6% respectively in understanding human preference for text-to-image synthesis.
• ImageReward aligns with human rankings and shows better distinguishability across models and samples compared to CLIP and FID baselines.
• ImageReward serves as a promising zero-shot automatic metric for evaluating and ranking text-to-image models based on real user prompts.
• ReFL directly fine-tunes diffusion models using ImageReward feedback, achieving the best performance in human evaluations against several baselines and alternative tuning methods.
• ReFL demonstrates advantages over data augmentation and loss-reweighting approaches by providing direct gradient feedback from the reward model.
• Examining ImageReward scores across denoising steps reveals that reliable human-preference signals emerge at later steps (around 30+ denoising steps), enabling effective late-step tuning.