[Paper Review] Automated Machine Learning Service Composition

Felix Mohr, Marcel Wever+1 more|arXiv (Cornell University)|Sep 3, 2018

Service-Oriented Architecture and Web Services16 references3 citations

TL;DR

This paper proposes MLS-Plan, a hierarchical planning-based algorithm for automated machine learning (Auto-ML) service composition that dynamically evaluates pipeline quality through execution rather than static quality-of-service metrics. It demonstrates that service-oriented composition—by combining algorithms from multiple frameworks (WEKA and scikit-learn)—yields significantly better performance than non-service-based approaches, especially in complex, real-world Auto-ML scenarios.

ABSTRACT

Automated service composition as the process of creating new software in an automated fashion has been studied in many different ways over the last decade. However, the impact of automated service composition has been rather small as its utility in real-world applications has not been demonstrated so far. This paper presents ool, an algorithm for automated service composition applied to the area of machine learning. Empirically, we show that ool is competitive and sometimes beats algorithms that solve the same task but not benefit of the advantages of a service model. Thereby, we present a real-world example that demonstrates the utility of automated service composition in contrast to non-service oriented solutions in the same area.

Motivation & Objective

• To demonstrate the practical utility of automated service composition in real-world machine learning applications, where prior approaches were limited to toy examples.
• To address the gap in automated service composition by enabling cross-framework integration of machine learning algorithms via a service abstraction layer.
• To evaluate whether service-based composition offers measurable advantages over non-service-oriented Auto-ML tools in terms of classification accuracy and solution diversity.
• To isolate the impact of service-oriented design from search strategy by comparing MLS-Plan with and without multi-framework access.
• To validate that execution-based quality evaluation is viable and effective in real-world Auto-ML pipelines where static QoS metrics are unavailable.

Proposed method

• MLS-Plan reduces the machine learning pipeline composition problem to hierarchical task planning (HTN), where complex tasks are recursively decomposed into atomic service operations.
• The planner uses a black-box objective function that evaluates candidate pipelines by executing them on benchmark data and measuring classification accuracy.
• Services from both WEKA and scikit-learn are exposed as first-class components, enabling cross-framework pipeline composition.
• The search process is guided by empirical performance feedback from executed pipelines, not static QoS attributes, allowing optimization of dynamic, data-dependent metrics.
• A custom planner is implemented to support iterative refinement and performance feedback, overcoming limitations of classical planners in handling dynamic quality evaluation.
• The approach supports both single-library (WEKA-only or scikit-learn-only) and multi-library (both) configurations to isolate the benefit of service composition.

Experimental results

Research questions

• RQ1 Can automated service composition outperform non-service-based Auto-ML tools in real-world classification tasks?
• RQ2 Does combining algorithms from multiple machine learning frameworks (e.g., WEKA and scikit-learn) via a service abstraction lead to better pipeline performance?
• RQ3 How does the increased search space from multi-framework access affect the performance and convergence of automated pipeline composition?
• RQ4 Is execution-based quality evaluation—measuring accuracy via benchmark runs—more effective than static QoS-based optimization in Auto-ML?
• RQ5 To what extent does the service-oriented architecture improve the diversity and quality of learned pipelines compared to monolithic framework solutions?

Key findings

• MLS-Plan with access to both WEKA and scikit-learn services significantly outperforms single-framework variants in many datasets, achieving the best average error rate in several cases.
• The service-based approach achieved statistically significant improvements over non-service-based counterparts in multiple datasets, demonstrating tangible benefits of cross-framework composition.
• In some cases, the multi-library version underperformed due to the larger search space requiring more time to explore effectively, indicating a trade-off between solution diversity and search efficiency.
• The overall results show that MLS-Plan with services is not universally dominant but is the best-performing option in a substantial number of real-world Auto-ML scenarios.
• The empirical evaluation confirms that execution-driven quality feedback enables effective optimization even when static quality metrics are unavailable or inapplicable.
• The study provides evidence that automated service composition is a viable and valuable technique for real-world Auto-ML, particularly when leveraging heterogeneous algorithm libraries through a unified service interface.