[Paper Review] Parity Models: A General Framework for Coding-Based Resilience in ML Inference

Jack Kosaian, K. V. Rashmi+1 more|arXiv (Cornell University)|May 2, 2019

Machine Learning and Data Classification84 references11 citations

TL;DR

ParM introduces a general framework that uses neural network-based parity models to enable coding-based resilience in machine learning inference systems, allowing fast reconstruction of unavailable predictions during slowdowns or failures. It reduces tail latency by up to 3.5× compared to resource-equivalent methods while using 2–4× less overhead than replication.

ABSTRACT

Machine learning models are becoming the primary workhorses for many applications. Production services deploy models through prediction serving systems that take in queries and return predictions by performing inference on machine learning models. In order to scale to high query rates, prediction serving systems are run on many machines in cluster settings, and thus are prone to slowdowns and failures that inflate tail latency and cause violations of strict latency targets. Current approaches to reducing tail latency are inadequate for the latency targets of prediction serving, incur high resource overhead, or are inapplicable to the computations performed during inference. We present ParM, a novel, general framework for making use of ideas from erasure coding and machine learning to achieve low-latency, resource-efficient resilience to slowdowns and failures in prediction serving systems. ParM encodes multiple queries together into a single parity query and performs inference on the parity query using a parity model. A decoder uses the output of a parity model to reconstruct approximations of unavailable predictions. ParM uses neural networks to learn parity models that enable simple, fast encoders and decoders to reconstruct unavailable predictions for a variety of inference tasks such as image classification, speech recognition, and object localization. We build ParM atop an open-source prediction serving system and through extensive evaluation show that ParM improves overall accuracy in the face of unavailability with low latency while using 2-4$ imes$ less additional resources than replication-based approaches. ParM reduces the gap between 99.9th percentile and median latency by up to $3.5 imes$ compared to approaches that use an equal amount of resources, while maintaining the same median.

Motivation & Objective

• Address the challenge of high tail latency in production prediction serving systems due to slowdowns and failures in distributed clusters.
• Overcome limitations of existing approaches—such as replication (high resource cost) and non-agnostic coded-computation methods (latency overhead or inapplicability to inference).
• Design a general-purpose framework that enables low-latency, resource-efficient resilience for diverse ML inference tasks including image classification, speech recognition, and object localization.
• Introduce parity models as a novel component to enable fast, task-specific encoding and decoding of queries for resilient inference.
• Achieve resilience that is agnostic to the cause of unavailability while maintaining high prediction accuracy during failures or slowdowns.

Proposed method

• Encode multiple input queries into a single parity query using a learnable encoder, enabling joint inference on a parity model.
• Train a neural network-based parity model to map the parity query to a representation that allows reconstruction of individual predictions.
• Use a decoder network to reconstruct approximations of unavailable predictions from the output of the parity model.
• Design task-specific encoders and decoders (e.g., concatenation-based for image classification) to optimize performance and reduce bandwidth overhead.
• Integrate ParM into an open-source prediction serving system to evaluate real-world performance and latency characteristics.
• Leverage erasure coding principles—where any k out of (k+r) units can reconstruct the original data—adapted to inference workloads via learned models.

Experimental results

Research questions

• RQ1 Can a general framework be designed to provide coding-based resilience in ML inference systems that is both low-latency and resource-efficient?
• RQ2 How can neural networks be used to learn parity models that enable fast, accurate reconstruction of predictions during slowdowns or failures?
• RQ3 To what extent can ParM reduce tail latency compared to replication and other coded-computation approaches while using fewer resources?
• RQ4 How does ParM perform across diverse inference tasks such as image classification, speech recognition, and object localization?
• RQ5 Can ParM maintain median latency while significantly reducing the 99.9th percentile latency gap without proactively degrading prediction accuracy?

Key findings

• ParM reduces the gap between 99.9th percentile and median latency by up to 3.5× compared to resource-equivalent approaches, while maintaining the same median latency.
• ParM uses 2–4× less additional resource overhead than replication-based approaches for achieving similar resilience.
• The framework improves overall prediction accuracy under unavailability by enabling fast reconstruction of failed or slow predictions.
• ParM’s approach incurs only 1/k network bandwidth overhead for k queries, compared to k× overhead in methods like image concatenation (e.g., Narra et al. 2019).
• ParM’s learned encoders and decoders enable fast, low-latency reconstruction without introducing latency when no failures occur.
• The framework is generalizable across diverse inference tasks, including image classification, speech recognition, and object localization, with task-specific optimizations.