[Paper Review] LIBRE: Learning Interpretable Boolean Rule Ensembles

Graziano Mita, Paolo Papotti+2 more|arXiv (Cornell University)|Nov 15, 2019

Explainable Artificial Intelligence (XAI)35 references5 citations

TL;DR

LIBRE is a novel ensemble method that learns interpretable Boolean rule sets by combining bottom-up, weak learners operating on random feature subsets, enabling strong generalization in both balanced and imbalanced data. It achieves competitive predictive performance with highly compact, human-readable rule sets, outperforming existing rule-based methods in interpretability and scalability while natively handling missing values.

ABSTRACT

We present a novel method - LIBRE - to learn an interpretable classifier, which materializes as a set of Boolean rules. LIBRE uses an ensemble of bottom-up weak learners operating on a random subset of features, which allows for the learning of rules that generalize well on unseen data even in imbalanced settings. Weak learners are combined with a simple union so that the final ensemble is also interpretable. Experimental results indicate that LIBRE efficiently strikes the right balance between prediction accuracy, which is competitive with black box methods, and interpretability, which is often superior to alternative methods from the literature.

Motivation & Objective

• To address the challenge of learning interpretable yet accurate classifiers in imbalanced data settings where traditional rule-based methods fail to capture minority classes.
• To overcome the limitations of existing rule learning methods—such as excessive rule counts, poor generalization, and sensitivity to noise and data fragmentation—by leveraging ensemble learning.
• To maintain interpretability through compact rule sets (few rules, short rule length) while achieving predictive performance competitive with black-box models.
• To enable native handling of missing values in both categorical and continuous features without requiring preprocessing, enhancing real-world applicability.
• To scale effectively to large datasets by reducing the computational burden through feature subsampling and efficient rule generation.

Proposed method

• LIBRE employs an ensemble of bottom-up weak learners that generate rules via monotone Boolean function synthesis, starting from individual examples and generalizing them based on coverage and accuracy.
• Each weak learner operates on a random subset of features, reducing overfitting and improving generalization, especially in imbalanced scenarios.
• Candidate rules from all weak learners are combined using a simple union operation, preserving interpretability by avoiding complex meta-models or weighting schemes.
• The method does not impose artificial constraints on rule size or support, allowing it to discover complex, meaningful patterns while maintaining compactness through ensemble averaging.
• Rule selection is guided by interpretability objectives, favoring small rule sets with minimal redundancy, which are evaluated via F1-score and rule count metrics.
• The framework natively supports missing values in both categorical and continuous features, eliminating the need for preprocessing steps like imputation or binarization.

Experimental results

Research questions

• RQ1 Can an ensemble of bottom-up rule learners improve generalization and robustness in imbalanced datasets while preserving interpretability?
• RQ2 How does LIBRE compare to state-of-the-art rule-based methods in terms of predictive performance, rule set compactness, and scalability?
• RQ3 To what extent can LIBRE handle missing values without requiring data preprocessing, and how does this affect performance?
• RQ4 Does combining bottom-up learners via union maintain interpretability while achieving performance comparable to black-box models?
• RQ5 How does LIBRE scale with increasing dataset size and feature dimensionality compared to existing rule learning methods?

Key findings

• LIBRE achieves F1-scores competitive with black-box models like XGBoost and Random Forest, especially in imbalanced settings where other rule-based methods fail.
• On the Liver and Adult datasets, LIBRE produces the most compact rule sets—fewer than 10 rules—while maintaining high F1-scores, outperforming ripper-k, s-brl, and brs in interpretability.
• For the Sap-Full dataset with extensive missing values, LIBRE maintains strong performance without preprocessing, unlike ripper-k, brs, and modlem, which require preprocessing for continuous features.
• In scalability tests, LIBRE completes training on 1 million records in under 400 seconds (399±8s), while modlem and brs fail with out-of-memory errors on the same data.
• LIBRE’s runtime scales sublinearly with dataset size, and its weak learners process only the positive and negative sets separately, reducing effective computation even for large datasets.
• The method produces highly interpretable models: on average, it generates fewer than 10 rules for large datasets like Adult and Bank, with minimal rule length inflation, unlike s-brl where rule complexity grows rapidly with rule count.