[论文解读] On-Device Machine Learning: An Algorithms and Learning Theory Perspective

Sauptik Dhar, Junyao Guo+4 more|arXiv (Cornell University)|Nov 2, 2019

Data Stream Mining Techniques参考文献 205被引用 45

一句话总结

本论文对边缘设备学习进行综述，将其框定为资源受限学习，核心资源为计算和内存，并讨论面向边缘设备的算法与学习理论，以及挑战与未来方向。

ABSTRACT

The predominant paradigm for using machine learning models on a device is to train a model in the cloud and perform inference using the trained model on the device. However, with increasing number of smart devices and improved hardware, there is interest in performing model training on the device. Given this surge in interest, a comprehensive survey of the field from a device-agnostic perspective sets the stage for both understanding the state-of-the-art and for identifying open challenges and future avenues of research. However, on-device learning is an expansive field with connections to a large number of related topics in AI and machine learning (including online learning, model adaptation, one/few-shot learning, etc.). Hence, covering such a large number of topics in a single survey is impractical. This survey finds a middle ground by reformulating the problem of on-device learning as resource constrained learning where the resources are compute and memory. This reformulation allows tools, techniques, and algorithms from a wide variety of research areas to be compared equitably. In addition to summarizing the state-of-the-art, the survey also identifies a number of challenges and next steps for both the algorithmic and theoretical aspects of on-device learning.

研究动机与目标

• 定义边缘设备，并将边内学习作为云端训练与推理的替代方案的动机。
• 将边内学习重新表述为以计算和内存为核心资源的资源受限学习。
• 调查在资源受限设备上训练的算法方法与理论考量。
• 识别资源高效的边缘设备学习中的挑战与未来研究方向。

提出的方法

• 将边内学习重新表述为以计算和内存为主要资源的资源受限学习。
• 调研用于边内学习的硬件、库、算法和理论体系及其相互作用。
• 按资源占用特征和实际分析对算法方法进行分类。
• 从资源度量（如 MACs/FLOPs、内存、激活）角度讨论传统模型和深度学习模型。
• 提出一个在不同设备上跨设备比较算法的框架，与具体硬件无关。

实验结果

研究问题

• RQ1 在计算和内存约束下，如何有效实现边内模型训练？
• RQ2 哪些关键的算法与理论发展能够推动资源高效的边内学习？
• RQ3 资源约束如何影响边缘设备学习方法的评估与选择？
• RQ4 资源高效的边内学习中的开放挑战与未来方向是什么？

主要发现

• 边内学习可以有效地被作为以计算和内存为核心资源的资源受限学习来研究。
• 把硬件感知指标（MACs/FLOPs、权重、激活）与传统机器学习算法一起用于评估DNN的资源占用。
• 边缘硬件种类繁多，需要轻量化模型和细致的分析来确保在设备上可行的训练。
• 当前对边缘设备上的推理以及在高性能平台上的训练有基准测试，突显了需要更深入的边内训练分析。
• 该综述概述了资源受限学习的理论框架，包括在资源限制下提供性能保证的新理论与挑战。