[Paper Review] On-Device Machine Learning: An Algorithms and Learning Theory Perspective

Sauptik Dhar, Junyao Guo+4 more|arXiv (Cornell University)|Nov 2, 2019

Data Stream Mining Techniques205 references45 citations

TL;DR

This paper surveys on-device learning by framing it as resource-constrained learning with two main resources: compute and memory, and discusses algorithms and learning theory for edge devices, along with challenges and future directions.

ABSTRACT

The predominant paradigm for using machine learning models on a device is to train a model in the cloud and perform inference using the trained model on the device. However, with increasing number of smart devices and improved hardware, there is interest in performing model training on the device. Given this surge in interest, a comprehensive survey of the field from a device-agnostic perspective sets the stage for both understanding the state-of-the-art and for identifying open challenges and future avenues of research. However, on-device learning is an expansive field with connections to a large number of related topics in AI and machine learning (including online learning, model adaptation, one/few-shot learning, etc.). Hence, covering such a large number of topics in a single survey is impractical. This survey finds a middle ground by reformulating the problem of on-device learning as resource constrained learning where the resources are compute and memory. This reformulation allows tools, techniques, and algorithms from a wide variety of research areas to be compared equitably. In addition to summarizing the state-of-the-art, the survey also identifies a number of challenges and next steps for both the algorithmic and theoretical aspects of on-device learning.

Motivation & Objective

• Define edge devices and motivate on-device learning as an alternative to cloud-based training and inference.
• Reformulate on-device learning as resource-constrained learning focusing on compute and memory.
• Survey algorithmic approaches and theoretical considerations for training on resource-limited devices.
• Identify challenges and future research directions in resource-efficient on-device learning.

Proposed method

• Reformulate on-device learning as resource-constrained learning with compute and memory as primary resources.
• Survey hardware, library, algorithm, and theory stack for on-device learning and their interactions.
• Categorize algorithmic approaches by resource footprint characterization and practical profiling.
• Discuss traditional and deep learning models in terms of resource metrics like MACs/FLOPs, memory, and activations.
• Present a framework for comparing algorithms across devices independent of specific hardware.

Experimental results

Research questions

• RQ1 How can on-device model training be effectively achieved under compute and memory constraints?
• RQ2 What are the key algorithmic and theoretical developments enabling resource-efficient on-device learning?
• RQ3 How do resource constraints influence the evaluation and selection of learning methods for edge devices?
• RQ4 What are the open challenges and future directions in resource-efficient on-device learning?

Key findings

• On-device learning can be fruitfully studied as resource-constrained learning using compute and memory as core resources.
• Hardware-aware metrics (MACs/FLOPs, weights, activations) are used to assess the resource footprint of DNNs, alongside traditional ML algorithms.
• There is a wide spectrum of edge hardware, requiring lightweight models and careful profiling to ensure feasibility of training on-device.
• Current benchmarking efforts exist for inference on edge devices and for training on high-performance platforms, highlighting the need for deeper on-device training profiling.
• The survey outlines theoretical frameworks for resource-constrained learning, including new theories and challenges in providing performance guarantees under resource limits.