[Paper Review] Efficient Online ML API Selection for Multi-Label Classification Tasks

Lingjiao Chen, Matei Zaharia+1 more|arXiv (Cornell University)|Feb 18, 2021

Text and Document Classification Technologies5 citations

TL;DR

This paper proposes FrugalMUT, a principled online framework for efficient multi-label classification API selection that adaptively combines predictions from multiple ML APIs to maximize accuracy within a user-defined budget. By estimating prediction reliability and using a fast sequential selector, FrugalMUT achieves over 90% cost reduction while matching or improving upon the best single API's accuracy across diverse tasks like image and text classification.

ABSTRACT

Multi-label classification tasks such as OCR and multi-object recognition are a major focus of the growing machine learning as a service industry. While many multi-label prediction APIs are available, it is challenging for users to decide which API to use for their own data and budget, due to the heterogeneity in those APIs' price and performance. Recent work shows how to select from single-label prediction APIs. However the computation complexity of the previous approach is exponential in the number of labels and hence is not suitable for settings like OCR. In this work, we propose FrugalMCT, a principled framework that adaptively selects the APIs to use for different data in an online fashion while respecting user's budget. The API selection problem is cast as an integer linear program, which we show has a special structure that we leverage to develop an efficient online API selector with strong performance guarantees. We conduct systematic experiments using ML APIs from Google, Microsoft, Amazon, IBM, Tencent and other providers for tasks including multi-label image classification, scene text recognition and named entity recognition. Across diverse tasks, FrugalMCT can achieve over 90% cost reduction while matching the accuracy of the best single API, or up to 8% better accuracy while matching the best API's cost.

Motivation & Objective

• To address the challenge of selecting and combining multiple heterogeneous ML APIs for multi-label classification tasks under strict budget constraints.
• To improve prediction accuracy beyond the best single API by intelligently combining outputs from multiple providers.
• To reduce computational cost in online settings where the number of labels is large, such as in multi-label image and text classification.
• To model and exploit correlations between API predictions to enhance overall reliability and accuracy.
• To design a scalable, real-time solution that dynamically selects the optimal combination of APIs based on input features and budget constraints.

Proposed method

• FrugalMUT employs a real-time, online framework that evaluates each API's prediction reliability on a per-input basis using an uncertainty estimator.
• It uses a fast sequential selector that balances accuracy and cost by selecting the optimal combination of APIs based on estimated reliability and budget constraints.
• The framework models the relative strengths of different API combinations using a learned reliability estimator, enabling dynamic adaptation to input-specific characteristics.
• It integrates predictions from multiple APIs by combining them in a way that preserves or enhances the true label set, even when individual APIs make disjoint predictions.
• The system supports both single-API and multi-API combinations, with a fallback mechanism that stops early if a high-confidence prediction is reached.
• It is designed to handle large label sets efficiently, avoiding the exponential complexity of brute-force API combination evaluation.

Experimental results

Research questions

• RQ1 How can we efficiently select and combine multiple ML APIs in real time to maximize prediction accuracy for multi-label classification tasks?
• RQ2 What is the optimal strategy for balancing cost and accuracy when multiple heterogeneous APIs are available with varying precision and pricing?
• RQ3 Can we model the correlation between API predictions to improve overall accuracy beyond the best single API?
• RQ4 How does the system perform under strict budget constraints across diverse input types and label sets?
• RQ5 Can the framework scale effectively to large multi-label problems without incurring prohibitive computational costs?

Key findings

• FrugalMUT achieves over 90% cost reduction compared to using the best single API, while maintaining or exceeding its accuracy.
• On multi-label image classification tasks, FrugalMUT matches the accuracy of the best single API but at a fraction of the cost.
• For tasks like named entity recognition and text extraction, FrugalMUT improves accuracy by up to 8% compared to the best individual API, while still reducing cost.
• The framework effectively models and leverages correlations between API predictions, such as when one API detects 'person' and another 'car', and combines them to recover the full true label set.
• In experiments with Google, Microsoft, Amazon, IBM, and Tencent APIs, FrugalMUT consistently outperformed single-API baselines in both cost efficiency and accuracy.
• The system demonstrates strong scalability, handling large label sets without the exponential computational cost seen in prior approaches.