[Paper Review] Boolean Decision Rules via Column Generation

Sanjeeb Dash, Oktay Günlük+1 more|arXiv (Cornell University)|May 24, 2018

Machine Learning and Data ClassificationComputer Science55 citations

TL;DR

The paper introduces a column generation approach to learn Boolean DNF/CNF rule sets for binary classification, optimizing accuracy-simplicity trade-offs without pre-mining rules, and demonstrates competitive performance on multiple datasets.

ABSTRACT

This paper considers the learning of Boolean rules in either disjunctive normal form (DNF, OR-of-ANDs, equivalent to decision rule sets) or conjunctive normal form (CNF, AND-of-ORs) as an interpretable model for classification. An integer program is formulated to optimally trade classification accuracy for rule simplicity. Column generation (CG) is used to efficiently search over an exponential number of candidate clauses (conjunctions or disjunctions) without the need for heuristic rule mining. This approach also bounds the gap between the selected rule set and the best possible rule set on the training data. To handle large datasets, we propose an approximate CG algorithm using randomization. Compared to three recently proposed alternatives, the CG algorithm dominates the accuracy-simplicity trade-off in 7 out of 15 datasets. When maximized for accuracy, CG is competitive with rule learners designed for this purpose, sometimes finding significantly simpler solutions that are no less accurate.

Motivation & Objective

• Develop an interpretable Boolean rule model (DNF/CNF) for binary classification.
• Formulate an integer programming problem that balances accuracy and rule complexity.
• Use column generation to search over an exponential clause space efficiently.
• Provide an approximate CG variant for large datasets with performance guarantees.
• Demonstrate empirical advantages over recent alternatives on multiple datasets.

Proposed method

• Formulate an IP to minimize Hamming loss with a complexity bound on the rule set.
• Apply column generation to iteratively add clauses (conjunctions/disjunctions) that improve the solution.
• Define a pricing problem that searches for the most improving missing clause using dual variables.
• Represent clauses as conjunctions of binary features with a cap on clause size D.
• Provide an exact CG approach for small datasets and an approximate CG with randomization for large datasets.
• Compare CG against Bayesian Rule Sets, AM/BCD, RIPPER, CART, and RF across 16 datasets.

Experimental results

Research questions

• RQ1 Can column generation efficiently search over all potential Boolean clauses without pre-mining?
• RQ2 How does the accuracy-simplicity trade-off of CG compare to recent rule-set learners on standard datasets?
• RQ3 What are the theoretical guarantees and practical performance for CG on small vs. large datasets?
• RQ4 Does the approximate CG with random sampling degrade accuracy significantly on large datasets?
• RQ5 Can CG produce simpler yet accurate rule sets suitable for interpretable models?

Key findings

• CG dominates several recent methods on 8 of 16 datasets in the accuracy-simplicity trade-off.
• When optimized for accuracy, CG is competitive with RIPPER and can produce simpler models with no loss in accuracy on some datasets.
• Exact optimality proofs hold for small datasets; for larger datasets a practical two-stage approach (approx Pricing Problem plus Restricted MIP) yields strong results.
• CG often yields significantly simpler rule sets that maintain or improve accuracy on larger datasets (e.g., banknote, magic, FICO).
• On tic-tac-toe and some larger datasets, CG finds exact or near-exact rule sets that other methods fail to match.