[论文解读] Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization

Bing-Jing Hsieh, Ping-Chun Hsieh+1 more|arXiv (Cornell University)|Jun 8, 2021

Machine Learning and Data Classification被引用 6

一句话总结

本文提出了一种基于强化学习的少样本获取函数学习（FSAF）框架，采用贝叶斯深度Q网络（DQN）来学习获取函数的分布，从而实现仅需极少元数据即可快速适应多种黑箱函数。通过引入Kullback-Leibler正则化、演示诱导先验以及来自贝叶斯模型无关元学习的元损失，FSAF有效缓解了过拟合问题，并在合成与真实世界优化任务中实现了最先进水平的遗憾性能。

ABSTRACT

Bayesian optimization (BO) conventionally relies on handcrafted acquisition functions (AFs) to sequentially determine the sample points. However, it has been widely observed in practice that the best-performing AF in terms of regret can vary significantly under different types of black-box functions. It has remained a challenge to design one AF that can attain the best performance over a wide variety of black-box functions. This paper aims to attack this challenge through the perspective of reinforced few-shot AF learning (FSAF). Specifically, we first connect the notion of AFs with Q-functions and view a deep Q-network (DQN) as a surrogate differentiable AF. While it serves as a natural idea to combine DQN and an existing few-shot learning method, we identify that such a direct combination does not perform well due to severe overfitting, which is particularly critical in BO due to the need of a versatile sampling policy. To address this, we present a Bayesian variant of DQN with the following three features: (i) It learns a distribution of Q-networks as AFs based on the Kullback-Leibler regularization framework. This inherently provides the uncertainty required in sampling for BO and mitigates overfitting. (ii) For the prior of the Bayesian DQN, we propose to use a demo policy induced by an off-the-shelf AF for better training stability. (iii) On the meta-level, we leverage the meta-loss of Bayesian model-agnostic meta-learning, which serves as a natural companion to the proposed FSAF. Moreover, with the proper design of the Q-networks, FSAF is general-purpose in that it is agnostic to the dimension and the cardinality of the input domain. Through extensive experiments, we demonstrate that the FSAF achieves comparable or better regrets than the state-of-the-art benchmarks on a wide variety of synthetic and real-world test functions.

研究动机与目标

• 为解决在贝叶斯优化中设计单一获取函数以在多种黑箱函数上均表现良好的挑战。
• 通过仅使用少量元数据（如1-shot）实现获取函数的快速适应，克服现有元学习方法依赖大规模元数据集的局限性。
• 通过不确定性感知的贝叶斯建模，缓解基于深度强化学习的获取函数中的过拟合问题。
• 开发一种通用的、与维度无关的获取函数，具备可微性，并兼容多种优化目标。

提出的方法

• 将获取函数形式化为Q函数，并使用深度Q网络（DQN）作为获取函数的可微分、参数化的代理模型。
• 提出DQN的贝叶斯变体，通过Kullback-Leibler正则化学习Q网络参数的分布，从而内在地建模不确定性并减少过拟合。
• 采用从预训练获取函数（如EI）导出的演示策略作为贝叶斯DQN的先验，以提升训练稳定性。
• 利用来自贝叶斯模型无关元学习（MAML）的元损失，联合优化DQN在任务间的初始化与适应过程。
• 设计Q网络架构以与输入维度和领域基数无关，从而实现广泛适用性。
• 在元数据支持的源任务分布上训练FSAF模型，然后在每个新任务上仅用少量样本进行微调。

实验结果

研究问题

• RQ1 基于深度强化学习的获取函数能否仅用少量样本有效适应新的黑箱函数？
• RQ2 在贝叶斯优化的少样本适应过程中，如何缓解基于深度Q网络的获取函数中的过拟合问题？
• RQ3 结合结构化先验和元学习损失的贝叶斯DQN是否能实现优于标准DQN或手工设计获取函数的泛化性能？
• RQ4 所提出的FSAF在多样化的合成与真实世界优化任务中，相较于最先进获取函数的性能提升程度如何？

主要发现

• FSAF在广泛范围的合成与真实世界测试函数上实现了与最先进基准相当或更优的遗憾性能，展现出强大的泛化能力。
• 所提出的结合KL正则化与基于演示的先验的贝叶斯DQN，相比标准DQN+MAML，显著减少了过拟合，后者虽在训练中表现强劲，但在测试中泛化能力差。
• FSAF在不同类型黑箱函数上均表现出稳健性能，包括不同平滑度、多峰性及噪声水平的函数。
• 贝叶斯MAML的元损失集成实现了快速且稳定的少样本适应，即使在元数据有限的情况下性能下降也极小。
• FSAF具备通用性且与维度无关，在低维与高维输入域上均表现良好，无需修改网络架构。