[Paper Review] A Review of Meta-level Learning in the Context of Multi-component, Multi-level Evolving Prediction Systems

Abbas Raza Ali, Marcin Budka+1 more|arXiv (Cornell University)|Jul 17, 2020

Data Stream Mining Techniques92 references4 citations

TL;DR

This paper reviews meta-level learning (MLL) for evolving, multi-component, multi-level prediction systems, proposing a framework to automate algorithm selection and system adaptation in non-stationary data environments. It leverages meta-features and meta-knowledge to enable intelligent, real-time recommendations across pre-processing, learning algorithms, and concept drift handling, significantly reducing reliance on manual tuning and expert intervention.

ABSTRACT

The exponential growth of volume, variety and velocity of data is raising the need for investigations of automated or semi-automated ways to extract useful patterns from the data. It requires deep expert knowledge and extensive computational resources to find the most appropriate mapping of learning methods for a given problem. It becomes a challenge in the presence of numerous configurations of learning algorithms on massive amounts of data. So there is a need for an intelligent recommendation engine that can advise what is the best learning algorithm for a dataset. The techniques that are commonly used by experts are based on a trial and error approach evaluating and comparing a number of possible solutions against each other, using their prior experience on a specific domain, etc. The trial and error approach combined with the expert's prior knowledge, though computationally and time expensive, have been often shown to work for stationary problems where the processing is usually performed off-line. However, this approach would not normally be feasible to apply to non-stationary problems where streams of data are continuously arriving. Furthermore, in a non-stationary environment, the manual analysis of data and testing of various methods whenever there is a change in the underlying data distribution would be very difficult or simply infeasible. In that scenario and within an on-line predictive system, there are several tasks where Meta-learning can be used to effectively facilitate best recommendations including 1) pre-processing steps, 2) learning algorithms or their combination, 3) adaptivity mechanisms and their parameters, 4) recurring concept extraction, and 5) concept drift detection.

Motivation & Objective

• Address the challenge of selecting optimal learning algorithms for evolving, non-stationary data streams in real time.
• Reduce dependency on manual, trial-and-error approaches and expert intervention in algorithm configuration.
• Develop a scalable, automated recommendation system for multi-component, multi-level predictive systems.
• Enable on-line adaptation through meta-learning techniques that track concept drift and system performance.
• Establish a comprehensive framework for meta-knowledge representation and retrieval across diverse learning problems.

Proposed method

• Extract meta-features (MFs) from datasets using descriptive, statistical, information-theoretic, landmarking, and model-based approaches.
• Construct a meta-knowledge (MK) database by associating MFs with performance metrics of previously applied learning algorithms.
• Use meta-learners to map new problem characteristics to the most suitable algorithms based on historical MK.
• Apply landmarking techniques to estimate algorithm performance using lightweight, fast learners as proxies.
• Utilize decision tree-based model features (e.g., depth, node count, shape, homogeneity) to represent structural problem characteristics.
• Integrate meta-features into a unified representation for real-time decision-making in evolving prediction systems.

Experimental results

Research questions

• RQ1 How can meta-level learning effectively recommend optimal learning algorithms for new, unseen datasets in non-stationary environments?
• RQ2 What meta-features are most predictive of learning algorithm performance across diverse data types and problem domains?
• RQ3 How can meta-knowledge be efficiently represented and retrieved to support real-time system adaptation?
• RQ4 In what ways can meta-learning reduce the need for manual algorithm tuning and expert intervention in evolving prediction systems?
• RQ5 How do different meta-feature extraction methods (e.g., landmarking vs. model-based) compare in predictive accuracy for algorithm selection?

Key findings

• Meta-features such as correlation statistics, entropy, mutual information, and landmarking performance provide strong predictive signals for algorithm selection.
• Model-based meta-features like tree depth, node distribution, and branch length effectively capture structural complexity and guide algorithm recommendation.
• Landmarking approaches significantly reduce computational cost by using fast proxy learners to predict full algorithm performance.
• The integration of multiple meta-feature types (descriptive, statistical, information-theoretic, model-based) improves the robustness and generalization of meta-learners.
• Meta-knowledge databases built from historical performance data enable accurate, real-time recommendations with minimal latency in dynamic environments.
• The framework reduces reliance on expert knowledge and manual tuning, enabling scalable deployment of evolving prediction systems across diverse domains.