[论文解读] Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation

Xiangyu Zhao, Peiyuan Zhang+8 more|arXiv (Cornell University)|Mar 12, 2026

Generative Adversarial Networks and Image Synthesis被引用 0

一句话总结

FIRM 引入面向任务的鲁棒奖励模型用于图像编辑和文本到图像生成，并提出基线-奖励（Base-and-Bonus）强化学习策略，结合 CME 与 QMA 以提升保真度与遵循指令的能力。

ABSTRACT

Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.

研究动机与目标

• 通过在强化学习中依赖可靠的奖励模型（评判器）来推动鲁棒、可信的图像编辑与生成。
• 开发用于编辑与生成的高质量、任务特定奖励数据集。
• 创建一个人工标注的基准，用于评估奖励模型与人类判断的一致性。
• 提出防止奖励被篡改的奖励设计，并在强化学习中平衡竞争目标。

提出的方法

• 构建两条数据流水线：FIRM-Edit（差异优先）与 FIRM-Gen（先计划再评分），以训练奖励模型 FIRM-Edit-8B 与 FIRM-Gen-8B。
• 构建 FIRM-Bench，一个用于编辑与生成评判的人工标注基准。
• 以 Qwen3-VL-8B-Instruct 初始化的奖励模型进行训练；在 FIRM-Bench 上评估与人类判断的一致性。
• 将奖励融入基线-奖励（Base-and-Bonus）强化学习策略中，引入用于编辑的一致性调制执行（CME）和用于生成的质量调制对齐（QMA）。
• 应用 DiffusionNFT 基于强化学习来优化受 FIRM 奖励引导的编辑器与生成器。
• 在多项基准测试中展示编辑与生成任务的显著性能提升。

Figure 1 : Comparison of image editing results across different methods. “w. FIRM-Edit-8B” indicates that FIRM-Edit-8B is adopted as the reward model during RL process.

实验结果

研究问题

• RQ1 如何设计奖励模型（评判器）以可靠评估图像编辑与图像生成的保真度？
• RQ2 面向任务的奖励流水线是否比通用奖励模型在与人类判断的一致性方面有提升？
• RQ3 基线-奖励策略是否能够缓解奖励黑客问题并在编辑与生成的强化学习中平衡竞争目标？
• RQ4 在标准基准上，受 FIRM 奖励引导的强化学习模型是否显著优于现有基线？

主要发现

• FIRM-Edit-8B 与 FIRM-Gen-8B 在 FIRM-Bench 上与人类判断的一致性优于开放源代码的大型语言/多模态模型及若干专有模型。
• 在编辑任务中，FIRM-Edit-8B 在所有评估基线中获得最低的执行误差（MAE）0.53 与一致性 MAE 0.73，总体 MAE 为 0.62（为所评估模型中最低）。
• 在生成任务中，FIRM-Gen-8B 显示出与更大模型相当的总体 MAE（0.51），在复杂提示下表现出色。
• CME 奖励设计有效降低奖励黑客行为并提升编辑性能，而 QMA 在指令遵循较强时提升生成质量。
• 受 FIRM 奖励引导的强化学习在 GEditBench、ImgEdit、GenEval、DPGBench、TIIF 与 UniGenBench++ 等基准任务上实现了最先进或高度具竞争力的结果。
• FIRM-RL 模型（如 FIRM-Qwen-Edit、FIRM-SD3.5）在相对较小的数据规模（例如编辑任务仅 2,400 份样本）下就显著优于基线，与更大的基线相比具有优势。

Figure 2 : Comparison of T2I generation results across different methods. “w. FIRM-Gen-8B” indicates that FIRM-Gen-8B is adopted as the reward model during RL process.