[Paper Review] Multi-label classification search space in the MEKA software

Alex G. C. de Sá, Alex A. Freitas+2 more|arXiv (Cornell University)|Nov 28, 2018

Text and Document Classification Technologies60 references4 citations

TL;DR

This paper proposes a formal context-free grammar to model the search space of multi-label classification (MLC) algorithms in the MEKA and WEKA frameworks, enabling systematic experimental evaluation of automated MLC methods. It integrates 26 MLC algorithms, 28 single-label classifiers, and 10 meta-algorithms with explicit constraints, dependencies, and hyper-parameter configurations, providing a comprehensive, executable search space for automated machine learning pipelines.

ABSTRACT

This supplementary material aims to describe the proposed multi-label classification (MLC) search spaces based on the MEKA and WEKA softwares. First, we overview 26 MLC algorithms and meta-algorithms in MEKA, presenting their main characteristics, such as hyper-parameters, dependencies and constraints. Second, we review 28 single-label classification (SLC) algorithms, preprocessing algorithms and meta-algorithms in the WEKA software. These SLC algorithms were also studied because they are part of the proposed MLC search spaces. Fundamentally, this occurs due to the problem transformation nature of several MLC algorithms used in this work. These algorithms transform an MLC problem into one or several SLC problems in the first place and solve them with SLC model(s) in a next step. Therefore, understanding their main characteristics is crucial to this work. Finally, we present a formal description of the search spaces by proposing a context-free grammar that encompasses the 54 learning algorithms. This grammar basically comprehends the possible combinations, the constraints and dependencies among the learning algorithms.

Motivation & Objective

• To define a comprehensive and executable search space for automated multi-label classification (MLC) methods using MEKA and WEKA.
• To model dependencies, constraints, and hyper-parameter configurations across 26 MLC algorithms and 28 single-label classifiers.
• To enable systematic experimental evaluation of MLC pipelines by formalizing the interplay between problem transformation, algorithm adaptation, and meta-algorithms.
• To support automated machine learning (AutoML) workflows by providing a structured, grammar-based representation of viable MLC configurations.

Proposed method

• Proposes a context-free grammar that formally describes the 54 learning algorithms (26 MLC, 28 SLC, 10 meta-algorithms) used in MEKA and WEKA.
• Defines production rules for problem transformation (PT) methods, including specialized rules for constrained algorithms like Bayesian Classifier Chains (BCC) and Population of MCC (PMCC).
• Introduces distinct grammar rules for algorithm adaptation (AA) methods, exemplified by ML-BPNN with hyper-parameters: number of epochs, hidden units, learning rate, and momentum.
• Models meta-algorithm integration through hierarchical rules (<META-MLC1>, <META-MLC2>, <META-MLC3>) that enforce constraints on which MLC algorithms can be combined with which meta-algorithms.
• Encodes hyper-parameter ranges using random generation functions (e.g., RANDOM-INT, RANDOM-REAL) to reflect realistic search space boundaries.
• Ensures validity by encoding known MEKA software constraints, such as BCC only being combinable with four specific meta-algorithms, preventing invalid configurations.

Experimental results

Research questions

• RQ1 How can the search space of multi-label classification algorithms in MEKA and WEKA be formally described to support automated experimental evaluation?
• RQ2 What are the key dependencies, constraints, and hyper-parameter configurations across MLC algorithms and their integration with meta-algorithms?
• RQ3 Which combinations of problem transformation, algorithm adaptation, and meta-algorithm methods are valid and executable within the MEKA framework?
• RQ4 How can a context-free grammar be used to model the complex interplay between MLC algorithms, single-label classifiers, and meta-algorithms in a way that reflects real software behavior?
• RQ5 What is the role of threshold calibration (e.g., PCut1, PCutL) in improving MLC performance, and how is it integrated into the search space?

Key findings

• The proposed grammar successfully models 54 learning algorithms (26 MLC, 28 SLC, 10 meta-algorithms) with explicit constraints and hyper-parameter ranges.
• Problem transformation methods (PT) account for 84.21% of MLC algorithms (16 out of 19), with distinct grammar rules for different constraint levels, such as BCC and PMCC.
• The ML-BPNN algorithm is the only algorithm adaptation (AA) method modeled, with four hyper-parameters: epochs (10–1000), hidden units (0.2–1.0 × attributes), learning rate (0.001–0.1), and momentum (0.2–0.8).
• Meta-algorithms like BaggingML and EnsembleML require specific parameter ranges: bag size percent (10–50) and attribute percent (10–100), with EnsembleML using a distinct range (52–72) for its bag size.
• The grammar enforces software-level constraints, such as BCC being combinable only with four meta-algorithms (SM, RSML, EM, CM), preventing invalid configurations.
• Threshold calibration via PCut1 and PCutL is formally integrated, with PCut1 as the default, improving prediction reliability by aligning predicted label cardinality with training data.