[Paper Review] ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries

Vivek Kumar Sharma, Praneeth Vepakomma+4 more|arXiv (Cornell University)|Oct 5, 2019

Data Stream Mining Techniques19 references4 citations

TL;DR

ExpertMatcher automates machine learning model selection for resource-constrained users by using autoencoders to match client data representations to the most relevant pre-trained models on a central server. It achieves 99.34% average accuracy in coarse-level dataset assignment and up to 84.36% in fine-grained class matching, enabling efficient inference without retraining or data sharing.

ABSTRACT

In this work we introduce ExpertMatcher, a method for automating deep learning model selection using autoencoders. Specifically, we are interested in performing inference on data sources that are distributed across many clients using pretrained expert ML networks on a centralized server. The ExpertMatcher assigns the most relevant model(s) in the central server given the client's data representation. This allows resource-constrained clients in developing countries to utilize the most relevant ML models for their given task without having to evaluate the performance of each ML model. The method is generic and can be beneficial in any setup where there are local clients and numerous centralized expert ML models.

Motivation & Objective

• To address the challenge of selecting appropriate pre-trained ML models for users in resource-constrained countries with limited computational power, data, or ML expertise.
• To enable remote clients to dynamically assign their data to the most relevant expert model without sharing raw data with the server.
• To develop a generic, modular, and efficient method for model selection that supports both coarse-level (dataset) and fine-grained (class) matching.
• To reduce reliance on training large multi-task models that suffer from catastrophic forgetting and poor performance across diverse tasks.

Proposed method

• ExpertMatcher uses a single-layer MLP autoencoder (784 → 128 → 784) to learn a compact, shared representation of each expert model’s training data.
• For coarse-level assignment, it computes reconstruction error (MSE) between the client’s input and the autoencoder’s output, assigning the model with the lowest error.
• For fine-grained class assignment, it uses cosine similarity between the bottleneck layer representations of the client data and the autoencoder-learned class-specific representations.
• The method is modular: new expert models can be added to the server without retraining the entire system.
• The client sends only the data representation (not raw data) to the server, preserving privacy.
• The approach is evaluated using 6 benchmark datasets, with model assignment based on either MSE (coarse) or cosine similarity (fine-grained).

Experimental results

Research questions

• RQ1 Can autoencoder-based representations effectively identify the most relevant pre-trained model for a given client input in a distributed setting?
• RQ2 How accurately can the method perform coarse-level dataset matching across diverse benchmark datasets with minimal computational overhead?
• RQ3 To what extent can the method support fine-grained class-level matching when models are trained on different class distributions?
• RQ4 How does the performance of the autoencoder-based matcher compare to direct classification (e.g., MLP with softmax) in dataset and class assignment tasks?
• RQ5 Can the method be generalized to real-world medical and imaging applications, such as non-line-of-sight and diabetic retinopathy detection?

Key findings

• ExpertMatcher achieved 99.34% average accuracy in coarse-level dataset assignment across six benchmark datasets, with individual accuracies exceeding 99% for MNIST, STL-10, and HAR.
• The autoencoder-based approach matched the performance of a direct MLP classifier (99.95% accuracy) in coarse-level assignment, despite being simpler and more generalizable.
• For fine-grained class assignment, the method achieved 84.36% accuracy on MNIST and 71.78% on NLOS, demonstrating effectiveness in identifying models trained on similar class distributions.
• The method successfully generalized to domain-specific tasks, achieving 96.49% accuracy on diabetic retinopathy and 99.92% on non-line-of-sight imaging datasets.
• Performance remained stable across two non-overlapping client data subsets, indicating robustness to data sampling variance.
• The approach enables efficient, privacy-preserving model selection without requiring data sharing or retraining of the central model hub.