[Paper Review] Dimension Reduction Using Rule Ensemble Machine Learning Methods: A Numerical Study of Three Ensemble Methods

Orianna DeMasi, Juan Meza+1 more|arXiv (Cornell University)|Aug 30, 2011

Machine Learning and Data Classification24 references3 citations

TL;DR

This paper proposes a rule ensemble machine learning method that combines decision trees with penalized regression to create interpretable, sparse models that rank feature importance and reduce dimensionality. Applied to binary and multi-class problems, including a supernova image dataset, the method reduced features from 39 to 21 while maintaining high predictive accuracy, outperforming standard bagging and boosting in interpretability and feature selection.

ABSTRACT

Ensemble methods for supervised machine learning have become popular due to their ability to accurately predict class labels with groups of simple, lightweight "base learners." While ensembles offer computationally efficient models that have good predictive capability they tend to be large and offer little insight into the patterns or structure in a dataset. We consider an ensemble technique that returns a model of ranked rules. The model accurately predicts class labels and has the advantage of indicating which parameter constraints are most useful for predicting those labels. An example of the rule ensemble method successfully ranking rules and selecting attributes is given with a dataset containing images of potential supernovas where the number of necessary features is reduced from 39 to 21. We also compare the rule ensemble method on a set of multi-class problems with boosting and bagging, which are two well known ensemble techniques that use decision trees as base learners, but do not have a rule ranking scheme.

Motivation & Objective

• To develop a machine learning method that combines predictive accuracy with model interpretability by transforming ensemble models into ranked rule sets.
• To address the lack of feature importance insight in traditional ensemble methods like bagging and boosting, which produce complex, opaque models.
• To extend the rule ensemble method to multi-class classification using one-vs-all strategies for broader applicability.
• To evaluate different coefficient-solving techniques in the rule ensemble framework for improved performance and stability.
• To demonstrate the method’s effectiveness in real-world scientific datasets, such as supernova image classification, by reducing high-dimensional feature spaces.

Proposed method

• The method generates rules from decision trees, where each rule corresponds to a hypercube in the feature space defined by indicator functions.
• Rules are combined using penalized regression (e.g., Lasso-like optimization) to create a sparse model that selects only the most predictive rules.
• The objective function minimizes a ramp loss function over training data while applying a penalty to reduce overfitting and enforce sparsity.
• The algorithm uses a coordinate descent approach with a fast gradient update strategy, where coefficients are updated stepwise based on the negative gradient of the loss.
• A two-case gradient update mechanism is employed: a cheap update when rule indicators remain unchanged, and a full recomputation when indicators flip due to coefficient updates.
• The method supports multi-class problems via a one-vs-all classification strategy, training separate rule ensembles for each class.

Experimental results

Research questions

• RQ1 Can rule ensemble methods effectively reduce dimensionality in high-dimensional datasets while preserving or improving predictive accuracy?
• RQ2 How does the rule ensemble method compare to standard bagging and boosting in terms of predictive performance and feature selection capability?
• RQ3 What is the impact of different coefficient-solving algorithms on the stability and interpretability of the resulting rule models?
• RQ4 To what extent can rule ensembles identify and rank the most relevant features in complex scientific datasets, such as those from astronomical image analysis?
• RQ5 Can the rule ensemble framework be successfully extended to multi-class classification problems with minimal performance degradation?

Key findings

• The rule ensemble method successfully reduced the number of relevant features from 39 to 21 in a supernova image classification dataset, significantly simplifying the model without sacrificing accuracy.
• The method achieved high predictive accuracy on multiple benchmark datasets from the UCI Machine Learning Repository, demonstrating robustness across diverse problem types.
• Penalized regression in the rule ensemble framework enabled automatic pruning of irrelevant rules, resulting in sparse, interpretable models with ranked feature importance.
• The rule ensemble outperformed standard bagging and boosting in interpretability, as it explicitly ranked which feature constraints were most predictive of class labels.
• The fast gradient update algorithm enabled efficient optimization, making the method scalable to large datasets despite the complexity of rule-based models.
• The one-vs-all extension allowed effective multi-class classification, maintaining the interpretability and sparsity of the rule-based model across multiple classes.