[论文解读] FlexiBO: Cost-Aware Multi-Objective Optimization of Deep Neural Networks.

Shahriar Iqbal, Jianhai Su+2 more|arXiv (Cornell University)|Jan 18, 2020

Advanced Multi-Objective Optimization Algorithms参考文献 43被引用 15

一句话总结

FlexiBO 是一种面向多目标神经架构搜索的成本感知贝叶斯优化框架，通过根据各目标的评估成本对其优先级进行排序，采用成本加权的帕累托超体积获取函数。在7种DNN架构上，其对真实帕累托前沿的贡献比最先进方法高出28.44%，多样性提升25.64%。

ABSTRACT

One of the key challenges in designing machine learning systems is to determine the right balance amongst several objectives, which also oftentimes are incommensurable and conflicting. For example, when designing deep neural networks (DNNs), one often has to trade-off between multiple objectives, such as accuracy, energy consumption, and inference time. Typically, there is no single configuration that performs equally well for all objectives. Consequently, one is interested in identifying Pareto-optimal designs. Although different multi-objective optimization algorithms have been developed to identify Pareto-optimal configurations, state-of-the-art multi-objective optimization methods do not consider the different evaluation costs attending the objectives under consideration. This is particularly important for optimizing DNNs: the cost arising on account of assessing the accuracy of DNNs is orders of magnitude higher than that of measuring the energy consumption of pre-trained DNNs. We propose FlexiBO, a flexible Bayesian optimization method, to address this issue. We formulate a new acquisition function based on the improvement of the Pareto hyper-volume weighted by the measurement cost of each objective. Our acquisition function selects the next sample and objective that provides maximum information gain per unit of cost. We evaluated FlexiBO on 7 state-of-the-art DNNs for object detection, natural language processing, and speech recognition. Our results indicate that, when compared to other state-of-the-art methods across the 7 architectures we tested, the Pareto front obtained using FlexiBO has, on average, a 28.44% higher contribution to the true Pareto front and achieves 25.64% better diversity.

研究动机与目标

• 为解决DNN优化中各冲突目标之间评估成本不平衡的问题，例如准确率评估成本高而能耗消耗评估成本低。
• 开发一种多目标优化框架，能够在搜索过程中考虑各目标测量成本的差异。
• 在成本约束下，提升帕累托最优DNN配置的质量与多样性。
• 通过优先选择高信息量、低成本的评估，实现对帕累托前沿的高效探索。

提出的方法

• 提出一种新型获取函数，通过各目标评估成本的倒数对帕累托超体积改进进行加权。
• 基于单位成本下预期信息增益最大化的准则，选择下一个评估点和目标。
• 通过将各目标特定的测量成本建模为超参数，将成本感知机制整合到贝叶斯优化中。
• 使用高斯过程对DNN配置在多个目标上的性能进行建模。
• 优化获取函数以在最小化总评估成本的同时，平衡探索与利用。
• 在真实世界的DNN任务（包括目标检测、自然语言处理和语音识别）中应用该框架，并利用预训练模型实现快速评估。

实验结果

研究问题

• RQ1 当多目标DNN优化中各目标的评估成本存在显著差异时，如何提升其效率？
• RQ2 成本感知的获取函数能否提升帕累托最优DNN配置的质量与多样性？
• RQ3 将帕累托改进按测量成本加权，对收敛性和性能有何影响？
• RQ4 在帕累托前沿质量与多样性方面，FlexiBO与最先进多目标优化方法相比表现如何？
• RQ5 成本感知的贝叶斯优化能否在不牺牲解质量的前提下，减少昂贵的准确率评估次数？

主要发现

• 在7种DNN架构上，FlexiBO对真实帕累托前沿的贡献比最先进方法高出28.44%。
• 与基线方法相比，该方法使帕累托前沿的多样性提升了25.64%。
• 成本加权的获取函数通过优先选择低成本、高信息量的评估，实现了更高效的搜索。
• FlexiBO在保持或提升解质量的同时，减少了对昂贵准确率评估的依赖。
• 该框架在多种DNN任务（包括目标检测、自然语言处理和语音识别）中均表现出一致的性能提升。
• 结果证实，在获取函数设计中考虑评估成本，可带来更优的帕累托前沿近似效果。