[论文解读] Reinforcement Learning for Generative AI: A Survey

Yuanjiang Cao, Quan Z. Sheng+2 more|arXiv (Cornell University)|Aug 28, 2023

Machine Learning and Data Classification被引用 10

一句话总结

对多模态跨越多领域的强化学习如何提升生成式AI的综合概览，包含统一的分类法与对挑战与趋势的讨论，包括LLMs和扩散模型。

ABSTRACT

Deep Generative AI has been a long-standing essential topic in the machine learning community, which can impact a number of application areas like text generation and computer vision. The major paradigm to train a generative model is maximum likelihood estimation, which pushes the learner to capture and approximate the target data distribution by decreasing the divergence between the model distribution and the target distribution. This formulation successfully establishes the objective of generative tasks, while it is incapable of satisfying all the requirements that a user might expect from a generative model. Reinforcement learning, serving as a competitive option to inject new training signals by creating new objectives that exploit novel signals, has demonstrated its power and flexibility to incorporate human inductive bias from multiple angles, such as adversarial learning, hand-designed rules and learned reward model to build a performant model. Thereby, reinforcement learning has become a trending research field and has stretched the limits of generative AI in both model design and application. It is reasonable to summarize and conclude advances in recent years with a comprehensive review. Although there are surveys in different application areas recently, this survey aims to shed light on a high-level review that spans a range of application areas. We provide a rigorous taxonomy in this area and make sufficient coverage on various models and applications. Notably, we also surveyed the fast-developing large language model area. We conclude this survey by showing the potential directions that might tackle the limit of current models and expand the frontiers for generative AI.

研究动机与目标

• 对强化学习如何在多领域提高生成式AI进行高层次、全面分析。
• 引入统一的分类法以组织生成建模中的 RL 方法。
• 讨论实际应用、挑战与机会，包括非微分设定和奖励设计。
• 突出新兴方向和 RL 集成生成系统的潜在未来路径。

提出的方法

• 对 RL 在生成式 AI 领域的文献进行综述和分类法开发以组织研究。
• 对生成任务中无模型与基于模型的 RL 的理论与实践讨论。
• 分析 RL 如何处理非微分组件与非 ML 训练信号。
• 对奖励设计方法的考察，包括判别器、手工设计规则、散度和数据驱动信号。
• 作为当前趋势的一部分，讨论与大语言模型和扩散模型的整合。

实验结果

研究问题

• RQ1 强化学习如何解决生成式 AI 中最大似然估计的局限性？
• RQ2 哪一个分类框架最能捕捉 RL 方法与生成模型在各应用中的交叉？
• RQ3 生成任务中 RL 的主要挑战及潜在解决方案是什么（如非微分性、稀疏奖励、长期信用）？
• RQ4 包括 LLMs 和基础模型在内的 RL 赋能生成系统的新兴方向和实际路径是什么？

主要发现

• 通过奖励函数提供灵活目标，使其对除训练数据分布之外的多样属性进行对齐。
• 通过对离散决策进行反向传播， RL 能在不可微的生成管线中进行学习。
• 多种 RL 方法（基于值、基于策略、演员-评论者、基于模型）可应用于生成设定，讨论了如 DQN、PPO、SAC、A3C 等方法。
• 基于判别器的和手工设计的奖励信号在引导生成方面被广泛使用，包括对抗和对比范式。
• 该综述强调将 RL 与大规模模型和扩散过程的整合作为一个关键的新兴趋势。