[论文解读] Identifying Co-Adaptation of Algorithmic and Implementational Innovations in Deep Reinforcement Learning: A Taxonomy and Case Study of Inference-based Algorithms.

Hiroki Furuta, Tadashi Kozuno+3 more|arXiv (Cornell University)|Mar 31, 2021

Reinforcement Learning in Robotics参考文献 58被引用 4

一句话总结

本文提出一种统一的基于推理的框架，用于在深度强化学习中将算法创新与实现细节解耦，采用单一的控制即推理目标。该研究识别出一系列共适应的实现选择（如tanh策略和网络规模），这些选择对性能至关重要，表明实现不匹配会导致显著的性能下降，而某些组件（如层归一化）则可在不同算法间迁移。

ABSTRACT

Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-agnostic and sometimes subtle. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements difficult. In this work, we focus on a series of inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.

研究动机与目标

• 为解决深度强化学习中算法与实现创新混杂的问题，该问题阻碍了严谨的性能分析。
• 系统性地将基于推理的演员-评论家算法中的算法新颖性与实现特定选择解耦。
• 识别出哪些实现细节与特定算法共适应，哪些可在不同算法间迁移。
• 提供一个统一的理论框架，基于优化原则（EM 与 KL 最小化）对算法进行分类。
• 通过明确性能提升的来源，实现更透明、可复现且可累积的深度强化学习进展。

提出的方法

• 推导出一个统一的控制即推理目标，涵盖 MPO、AWR 和 SAC，实现一致的比较。
• 根据算法是否使用期望最大化（EM）或直接最小化 Kullback-Leibler（KL）散度，对每个算法进行分类。
• 将所有其他规范（如激活函数、归一化方法和网络架构）视为实现细节。
• 通过系统性地在不同算法类型中改变实现组件，开展广泛的消融研究。
• 通过受控实验，隔离单个实现选择对最终性能的影响。
• 利用统一框架分析实现细节如何与算法设计选择共同演化。

实验结果

研究问题

• RQ1 在基于推理的强化学习算法中，哪些实现细节与特定的算法选择共适应？
• RQ2 在不导致性能下降的前提下，实现组件在不同算法间可迁移的程度如何？
• RQ3 算法区分（EM 与 KL 最小化）在多大程度上影响性能对实现选择的敏感性？
• RQ4 实现细节不匹配对算法性能有何影响？
• RQ5 哪些实现组件能在多个算法中提供一致增益，表明其具有通用实用性？

主要发现

• 将实现细节与算法类型错配会导致显著的性能下降，表明存在强烈的共适应性。
• tanh 策略和网络规模与特定算法类型高度共适应，不应随意迁移。
• 层归一化和 ELU 激活函数对 MPO 性能至关重要，并在 SAC 中也带来明显增益，表明具有可迁移优势。
• 批量归一化和 ReLU 等实现组件在不同算法间未表现出一致增益，提示其通用性有限。
• 统一的基于推理的框架成功揭示了算法设计与实现选择之间的隐藏依赖关系。
• 部分实现选择（如 ELU 和层归一化）在不同算法族中均提供一致的性能提升，表明具有广泛复用潜力。