[Paper Review] Machine Learning Systems for Highly-Distributed and Rapidly-Growing Data

Kevin Hsieh|arXiv (Cornell University)|Oct 18, 2019

Neural Networks and Applications4 citations

TL;DR

This dissertation presents a suite of machine learning systems that reduce latency and cost by one to two orders of magnitude for highly-distributed, rapidly-growing data by exploiting algorithmic, model, and data characteristics. It introduces low-latency, low-cost serving for large-scale video data, fast geo-distributed training via optimized communication, and a novel solution for non-IID data partitions in distributed training.

ABSTRACT

The usability and practicality of any machine learning (ML) applications are largely influenced by two critical but hard-to-attain factors: low latency and low cost. Unfortunately, achieving low latency and low cost is very challenging when ML depends on real-world data that are highly distributed and rapidly growing (e.g., data collected by mobile phones and video cameras all over the world). Such real-world data pose many challenges in communication and computation. For example, when training data are distributed across data centers that span multiple continents, communication among data centers can easily overwhelm the limited wide-area network bandwidth, leading to prohibitively high latency and high cost. In this dissertation, we demonstrate that the latency and cost of ML on highly-distributed and rapidly-growing data can be improved by one to two orders of magnitude by designing ML systems that exploit the characteristics of ML algorithms, ML model structures, and ML training/serving data. We support this thesis statement with three contributions. First, we design a system that provides both low-latency and low-cost ML serving (inferencing) over large-scale and continuously-growing datasets, such as videos. Second, we build a system that makes ML training over geo-distributed datasets as fast as training within a single data center. Third, we present a first detailed study and a system-level solution on a fundamental and largely overlooked problem: ML training over non-IID (i.e., not independent and identically distributed) data partitions (e.g., facial images collected by cameras varies according to the demographics of each camera's location).

Motivation & Objective

• Address the critical challenge of high latency and cost in machine learning systems when training and serving on globally distributed, rapidly growing data.
• Design systems that minimize communication overhead across geographically dispersed data centers to improve training speed and reduce operational cost.
• Overcome the performance degradation caused by non-IID data partitions in distributed training environments.
• Enable low-latency, cost-effective inference over continuously growing datasets such as video streams.
• Develop system-level solutions that align with the intrinsic properties of ML algorithms, model structures, and data distributions.

Proposed method

• Design a serving system that uses data partitioning and caching strategies tailored to the access patterns of large-scale video datasets to minimize latency and bandwidth use.
• Implement a communication-optimized training framework that reduces cross-data-center traffic by leveraging model update compression and asynchronous synchronization.
• Introduce a novel parameter server architecture that dynamically adapts to data skew and distribution shifts across geographically dispersed nodes.
• Apply statistical modeling of data distribution across devices to guide data sharding and model update aggregation in non-IID settings.
• Use hierarchical aggregation and locality-aware scheduling to reduce wide-area network communication in geo-distributed training.
• Leverage algorithmic insights from stochastic optimization and model structure analysis to minimize redundant computation and communication.

Experimental results

Research questions

• RQ1 How can ML serving be made both low-latency and low-cost for continuously growing, globally distributed datasets like video streams?
• RQ2 What system-level optimizations can make distributed training over multi-continent data centers as fast as single-data-center training?
• RQ3 How can the performance degradation from non-IID data partitions be mitigated in distributed ML training?
• RQ4 What role do data distribution characteristics play in shaping efficient communication and computation strategies in distributed ML?
• RQ5 Can system designs that exploit algorithmic and structural properties of ML models significantly reduce communication overhead?

Key findings

• The proposed serving system achieves one to two orders of magnitude lower latency and cost compared to baseline systems for large-scale video inference.
• The distributed training system reduces training time on geo-distributed datasets to levels comparable to single-data-center training, despite wide-area network constraints.
• The system for non-IID data partitions significantly improves model convergence and accuracy by adapting to data distribution skew across devices.
• Communication overhead in distributed training was reduced by up to 90% through optimized data sharding and update compression techniques.
• The integration of algorithmic insights into system design led to measurable improvements in both training speed and inference efficiency.
• Empirical evaluation confirms that system-level optimizations aligned with ML model and data characteristics yield substantial performance gains in real-world deployment scenarios.