[Paper Review] A flexible framework for communication-efficient machine learning: from HPC to IoT

Sarit Khirirat, Sindri Magnússon+2 more|arXiv (Cornell University)|Mar 13, 2020

IoT and Edge/Fog Computing17 references4 citations

TL;DR

This paper proposes a Communication-aware Adaptive Tuning (CAT) framework that dynamically adjusts gradient compression levels in machine learning to maximize objective improvement per communicated bit. By modeling platform-specific communication costs and optimizing compression per iteration, CAT achieves up to 5 orders of magnitude reduction in communication cost compared to full gradient descent, outperforming static tuning methods across HPC and IoT systems.

ABSTRACT

With the increasing scale of machine learning tasks, it has become essential to reduce the communication between computing nodes. Early work on gradient compression focused on the bottleneck between CPUs and GPUs, but communication-efficiency is now needed in a variety of different system architectures, from high-performance clusters to energy-constrained IoT devices. In the current practice, compression levels are typically chosen before training and settings that work well for one task may be vastly suboptimal for another dataset on another architecture. In this paper, we propose a flexible framework which adapts the compression level to the true gradient at each iteration, maximizing the improvement in the objective function that is achieved per communicated bit. Our framework is easy to adapt from one technology to the next by modeling how the communication cost depends on the compression level for the specific technology. Theoretical results and practical experiments indicate that the automatic tuning strategies significantly increase communication efficiency on several state-of-the-art compression schemes.

Motivation & Objective

• To address the communication bottleneck in distributed machine learning across diverse architectures, from high-performance clusters to energy-constrained IoT devices.
• To overcome the limitations of static compression tuning, which is suboptimal across different datasets and system configurations.
• To develop a flexible, technology-aware framework that adapts compression levels in real time based on gradient information and communication cost models.
• To improve communication efficiency by maximizing the ratio of objective function improvement to communication cost per iteration.

Proposed method

• The CAT framework models communication cost as a function of compression level, using either analytical models or empirical measurements for different hardware platforms.
• It formulates an optimization problem to select the compression level T that maximizes the ratio of guaranteed descent in the objective function to communication cost.
• For each compression scheme—sparsification, sparsification+quantization, and stochastic sparsification—the authors derive problem-specific descent lemmas to relate function improvement to the tuning parameter T.
• The framework is applied in multi-node settings, with convergence analysis provided for stochastic gradient descent under the stochastic sparsification scheme.
• A hybrid heuristic is introduced to reduce sparsification overhead by updating the sparsity budget every S iterations instead of every iteration.
• Communication cost models are validated experimentally using ZMQ-based TCP transmission, showing affine behavior suitable for modeling.

Experimental results

Research questions

• RQ1 How can gradient compression be dynamically adapted to both the data-dependent information content and platform-specific communication costs?
• RQ2 What is the optimal compression level T that maximizes the improvement in the objective function per communicated bit across different system architectures?
• RQ3 Can a unified framework achieve superior communication efficiency compared to static or heuristic tuning across diverse systems, from HPC to IoT?
• RQ4 How does the framework perform under varying communication models, such as payload-only versus packet-based models?

Key findings

• CAT reduces communication cost by up to 5 orders of magnitude compared to full gradient descent on the RCV1 dataset, significantly outperforming static tuning methods.
• In the packet communication model, CAT achieves roughly twice the communication efficiency of the dynamic tuning rule from Alistarh et al. (2017), which is agnostic to actual communication costs.
• The hybrid heuristic with S=200 reduces sparsification time by an order of magnitude while maintaining nearly identical convergence and communication cost performance.
• Experiments confirm that an affine model accurately captures end-to-end transmission time across different sparsity budgets, validating the communication cost modeling approach.
• For stochastic sparsification, CAT adaptively selects the optimal T even when the payload model suggests small T, demonstrating robustness to model assumptions.
• Despite higher sparsification time per iteration, CAT S+Q achieves faster wall-clock convergence due to drastically reduced communication overhead.