[Paper Review] Collage Inference: Using Coded Redundancy for Low Variance Distributed Image Classification

Krishna Giri Narra, Zhifeng Lin+3 more|arXiv (Cornell University)|Apr 27, 2019

Advanced Neural Network Applications42 references4 citations

TL;DR

This paper proposes collage inference, a coded redundancy technique using a custom convolutional neural network (collage-cnn) that classifies multiple images simultaneously in a single inference pass to reduce tail latency in distributed image classification. By treating multiple requests as a collective and using a low-cost, multi-image classifier as a backup, the method reduces 99th percentile latency by 1.2x to 2x and variance by 1.8x to 15x compared to replication, while maintaining high accuracy.

ABSTRACT

MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a cloud deployment due to uncertainties in resource allocations across many virtual instances. We propose the collage inference technique which uses a novel convolutional neural network model, collage-cnn, to provide low-cost redundancy. A collage-cnn model takes a collage image formed by combining multiple images and performs multi-image classification in one shot, albeit at slightly lower accuracy. We augment a collection of traditional single image classifier models with a single collage-cnn classifier which acts as their low-cost redundant backup. Collage-cnn provides backup classification results if any single image classification requests experience slowdown. Deploying the collage-cnn models in the cloud, we demonstrate that the 99th percentile tail latency of inference can be reduced by 1.2x to 2x compared to replication based approaches while providing high accuracy. Variation in inference latency can be reduced by 1.8x to 15x.

Motivation & Objective

• To address high tail latency and variance in cloud-based distributed image classification due to resource contention and straggler effects.
• To design a low-cost, scalable redundancy mechanism that avoids the high overhead of request replication in MLaaS environments.
• To develop a specialized CNN architecture (collage-cnn) capable of multi-image classification with minimal accuracy loss for use as a backup system.
• To evaluate the effectiveness of collage inference in reducing inference latency and variance without sacrificing prediction accuracy in real cloud deployments.

Proposed method

• The collage-cnn model processes a collage image formed by spatially combining multiple input images into a single multi-object input for joint classification.
• A single collage-cnn acts as a coded redundancy backup for multiple individual image classifiers (s-cnn models), providing fallback predictions if any individual request slows down.
• The method leverages the load balancer’s ability to group multiple inference requests into a collective, enabling efficient batch processing and redundancy with minimal added cost.
• Training data for the collage-cnn is generated using data augmentation and knowledge distillation techniques, with the s-cnn (ResNet-34) used as a teacher model to improve performance.
• The architecture is optimized for multi-object classification, achieving higher accuracy than standard parity models used in similar frameworks like ParM.
• The system is deployed in cloud environments to evaluate real-world performance, comparing latency and variance against replication-based approaches.

Experimental results

Research questions

• RQ1 Can a single multi-image classifier be used effectively as a low-cost backup to reduce tail latency in distributed image classification?
• RQ2 How does the accuracy of a collage-cnn compare to standard parity models when used for redundant inference?
• RQ3 To what extent can collage inference reduce 99th percentile latency and variance compared to request replication?
• RQ4 What is the trade-off between accuracy, resource cost, and latency reduction when using coded redundancy in MLaaS inference?

Key findings

• Collage inference reduces 99th percentile inference latency by 1.2x to 2x compared to replication-based approaches in cloud deployments.
• The method reduces latency variance by 1.8x to 15x, significantly improving consistency in response times.
• A 3x3 collage-cnn achieves 78.3% accuracy on CIFAR-10, outperforming a similarly sized student model (57%) and a ParM parity model (74%) for the same task.
• The collage-cnn model achieves higher accuracy than ParM’s parity model (88.91% vs. 74%) when classifying four CIFAR-10 images, despite using the same compute resources.
• Knowledge distillation with a ResNet-34 teacher model improves the accuracy of compressed models, but the custom collage-cnn still outperforms them.
• The approach enables effective straggler mitigation without the high compute and storage overhead of proactive replication or the accuracy loss of approximation techniques.