[Paper Review] DivBO: Diversity-aware CASH for Ensemble Learning

Yu Shen, Yu‐peng Lu+4 more|arXiv (Cornell University)|Feb 7, 2023

Machine Learning and Data Classification4 citations

TL;DR

DivBO is a Bayesian optimization framework for CASH (Combined Algorithm Selection and Hyperparameter optimization) that explicitly incorporates model diversity into the search process. By introducing a diversity surrogate and a weighted acquisition function, DivBO improves ensemble performance, achieving the best average rank (1.82 on validation, 1.73 on test) across 15 datasets compared to 10 baselines, including post-hoc ensemble methods in recent AutoML systems.

ABSTRACT

The Combined Algorithm Selection and Hyperparameters optimization (CASH) problem is one of the fundamental problems in Automated Machine Learning (AutoML). Motivated by the success of ensemble learning, recent AutoML systems build post-hoc ensembles to output the final predictions instead of using the best single learner. However, while most CASH methods focus on searching for a single learner with the best performance, they neglect the diversity among base learners (i.e., they may suggest similar configurations to previously evaluated ones), which is also a crucial consideration when building an ensemble. To tackle this issue and further enhance the ensemble performance, we propose DivBO, a diversity-aware framework to inject explicit search of diversity into the CASH problems. In the framework, we propose to use a diversity surrogate to predict the pair-wise diversity of two unseen configurations. Furthermore, we introduce a temporary pool and a weighted acquisition function to guide the search of both performance and diversity based on Bayesian optimization. Empirical results on 15 public datasets show that DivBO achieves the best average ranks (1.82 and 1.73) on both validation and test errors among 10 compared methods, including post-hoc designs in recent AutoML systems and state-of-the-art baselines for ensemble learning on CASH problems.

Motivation & Objective

• Address the lack of diversity in post-hoc ensemble learning within CASH frameworks, where AutoML systems often suggest highly similar configurations.
• Improve ensemble performance by explicitly balancing predictive performance and diversity during hyperparameter search.
• Develop a method that ensures base learners in the ensemble are both accurate and diverse, avoiding redundancy in predictions.
• Introduce a surrogate model to predict pairwise diversity between unseen configurations, enabling informed search decisions.
• Demonstrate that diversity-aware search leads to more effective ensembles without degrading individual model performance.

Proposed method

• Propose a diversity surrogate model to predict the pairwise diversity between two unseen machine learning configurations using regularized Euclidean distance of their probabilistic predictions.
• Integrate the diversity surrogate with a Bayesian optimization framework using a weighted acquisition function that balances performance and diversity objectives.
• Maintain a temporary pool of base learners during optimization to track ensemble evolution and guide search toward diverse, high-performing configurations.
• Use a saturating weight mechanism to dynamically balance the influence of performance and diversity surrogates in the acquisition function.
• Train the diversity surrogate incrementally using observed configurations and their pairwise diversity, improving prediction accuracy over time.
• Apply the framework within a CASH setting where the goal is to minimize validation error while maximizing diversity among selected models.

Experimental results

Research questions

• RQ1 Can explicit diversity search in CASH improve the performance of post-hoc ensembles compared to standard AutoML methods that prioritize only performance?
• RQ2 How well can a learned diversity surrogate predict actual pairwise diversity between model configurations compared to performance surrogates?
• RQ3 Does incorporating diversity into the Bayesian optimization acquisition function lead to more diverse and effective ensembles without sacrificing predictive accuracy?
• RQ4 How does the diversity of suggested configurations evolve over time, and does it correlate with improved ensemble generalization?
• RQ5 Can the diversity surrogate enable the system to avoid redundant configurations and improve ensemble diversity more effectively than baseline methods?

Key findings

• DivBO achieved the best average rank of 1.82 on validation error and 1.73 on test error across 15 public datasets, outperforming 10 compared methods including recent AutoML systems and state-of-the-art ensemble baselines.
• The diversity surrogate showed a stronger correlation with ground-truth diversity than the performance surrogate, indicating its reliability in guiding search.
• DivBO achieved a minimum diversity of 0.23 for suggested configurations after 200 iterations on the quake dataset, significantly higher than BO-ES and RB-ES (both ~0.05), indicating reduced redundancy.
• The effective pool update frequency in DivBO was 1.5 times per 50 iterations on average in the final phase, outperforming BO-ES (0.6) and RB-ES (0.8), showing more frequent and meaningful ensemble improvements.
• The final ensembles from DivBO exhibited an average pair-wise predictive disagreement of 0.27, substantially higher than RB-ES (0.13) and BO-ES (0.09), confirming greater diversity among base learners.
• Despite emphasizing diversity, DivBO maintained competitive base learner performance, with average validation error of 44.33% ± 2.67%, comparable to BO-ES (44.03%) and RB-ES (43.85%), proving no degradation in individual model quality.