[Paper Review] Trends and Advancements in Deep Neural Network Communication

Felix Sattler, Thomas Wiegand+1 more|arXiv (Cornell University)|Mar 6, 2020

Adversarial Robustness in Machine Learning91 references4 citations

TL;DR

This paper reviews communication-efficient deep neural network (DNN) training and inference in distributed, privacy-sensitive environments—particularly mobile and IoT systems—by shifting intelligence to the data via model parameter exchange. It surveys compression techniques, convergence challenges, and privacy-robustness trade-offs, advocating for unified research on communication efficiency across federated learning, on-device inference, and model transfer.

ABSTRACT

Due to their great performance and scalability properties neural networks have become ubiquitous building blocks of many applications. With the rise of mobile and IoT, these models now are also being increasingly applied in distributed settings, where the owners of the data are separated by limited communication channels and privacy constraints. To address the challenges of these distributed environments, a wide range of training and evaluation schemes have been developed, which require the communication of neural network parametrizations. These novel approaches, which bring the "intelligence to the data" have many advantages over traditional cloud solutions such as privacy-preservation, increased security and device autonomy, communication efficiency and high training speed. This paper gives an overview over the recent advancements and challenges in this new field of research at the intersection of machine learning and communications.

Motivation & Objective

• Address the limitations of centralized cloud-based machine learning (Cloud ML), including privacy risks, data ownership issues, and communication inefficiency in mobile and IoT environments.
• Identify the need for a new paradigm—Embedded ML—where models are communicated instead of raw data to preserve data locality and reduce bandwidth usage.
• Systematically analyze neural data compression techniques to reduce communication overhead in distributed DNN training and inference.
• Examine key challenges such as convergence under non-iid data, privacy leakage via model updates, and robustness against adversarial attacks in communication-efficient settings.
• Advocate for integrated research on communication efficiency, privacy, and robustness in distributed deep learning, especially for real-world deployment in 5G and IoT ecosystems.

Proposed method

• Categorize and compare four core settings of DNN communication: federated learning, on-device inference, model transfer, and collaborative training, all centered on exchanging neural network parameters.
• Review a wide range of model compression techniques—including quantization, pruning, and knowledge distillation—to reduce the size of transmitted DNN parameters.
• Analyze convergence properties of distributed training under communication compression, especially under non-iid (non-independent and identically distributed) data assumptions.
• Evaluate privacy-preserving mechanisms such as differential privacy, secure multi-party computation, and trusted execution environments, and their integration with compression methods.
• Assess synchronization strategies (synchronous vs. asynchronous training) and their impact on convergence speed and staleness of model updates.
• Highlight the role of standardization (e.g., MPEG Part 17) in enabling interoperable neural data transport and large-scale deployment of Embedded ML.

Experimental results

Research questions

• RQ1 How can communication efficiency be maximized in distributed deep learning without compromising model accuracy or convergence?
• RQ2 What are the trade-offs between model compression, privacy guarantees, and robustness in federated and on-device learning?
• RQ3 How do non-iid data distributions affect the convergence of compressed communication in federated learning?
• RQ4 To what extent can privacy-preserving techniques like differential privacy be effectively combined with neural data compression?
• RQ5 What role do communication protocols and standardization (e.g., MPEG Part 17) play in enabling scalable, interoperable Embedded ML systems?

Key findings

• Neural network parametrization communication is now a major bottleneck in distributed AI, especially in mobile and IoT settings with limited bandwidth and privacy constraints.
• Model compression techniques such as quantization and pruning significantly reduce the size of transmitted DNN parameters, improving communication efficiency without major accuracy loss.
• Convergence of distributed training under compression is sensitive to data non-iidness, and existing theoretical guarantees are less robust when compression is applied.
• Privacy-preserving methods like differential privacy and secure computation can leak information through model updates, necessitating careful integration with compression to avoid trade-offs.
• Asynchronous training avoids straggler delays but introduces staleness, which can slow convergence, especially in later training stages.
• Standardization efforts like MPEG Part 17 are emerging to define interoperable formats for neural data transport, enabling broader deployment of Embedded ML.