[Paper Review] Scalable Rule-Based Representation Learning for Interpretable Classification

Zhuo Wang, Wei Zhang+2 more|arXiv (Cornell University)|Sep 30, 2021

Machine Learning and Data Classification40 references20 citations

TL;DR

This paper proposes Rule-based Representation Learner (RRL), a scalable, interpretable classifier that learns discrete, non-fuzzy rules end-to-end using a novel training method called Gradient Grafting, which enables effective backpropagation through discrete model structures. RRL achieves state-of-the-art performance on both small and large datasets while maintaining high model interpretability through learned logical rules and improved logical activation functions for continuous feature discretization.

ABSTRACT

Rule-based models, e.g., decision trees, are widely used in scenarios demanding high model interpretability for their transparent inner structures and good model expressivity. However, rule-based models are hard to optimize, especially on large data sets, due to their discrete parameters and structures. Ensemble methods and fuzzy/soft rules are commonly used to improve performance, but they sacrifice the model interpretability. To obtain both good scalability and interpretability, we propose a new classifier, named Rule-based Representation Learner (RRL), that automatically learns interpretable non-fuzzy rules for data representation and classification. To train the non-differentiable RRL effectively, we project it to a continuous space and propose a novel training method, called Gradient Grafting, that can directly optimize the discrete model using gradient descent. An improved design of logical activation functions is also devised to increase the scalability of RRL and enable it to discretize the continuous features end-to-end. Exhaustive experiments on nine small and four large data sets show that RRL outperforms the competitive interpretable approaches and can be easily adjusted to obtain a trade-off between classification accuracy and model complexity for different scenarios. Our code is available at: https://github.com/12wang3/rrl.

Motivation & Objective

• To address the scalability and interpretability trade-off in rule-based models for large-scale data.
• To enable effective training of non-differentiable discrete rule models using gradient-based optimization.
• To design improved logical activation functions that support end-to-end discretization of continuous features.
• To develop a hierarchical rule-based model that supports flexible rule representation and rule importance evaluation.
• To validate RRL's performance and interpretability across small and large datasets, demonstrating its adaptability to different complexity-accuracy trade-offs.

Proposed method

• Proposes Rule-based Representation Learner (RRL), a hierarchical model that learns interpretable rules in conjunctive and disjunctive normal forms for data representation and classification.
• Introduces Gradient Grafting, a novel training method that enables direct gradient descent optimization of discrete RRL models by leveraging gradient information at both continuous and discrete parametric points.
• Designs improved logical activation functions that enhance scalability and enable end-to-end discretization of continuous features through a tailored binarization layer.
• Employs a linear layer with learnable rule weights to evaluate rule importance and support model interpretability.
• Uses L2 regularization to control model complexity, allowing users to trade off accuracy and interpretability by adjusting the regularization coefficient λ.
• Applies the model to both structured tabular data and evaluates performance across nine small and four large datasets.

Experimental results

Research questions

• RQ1 Can a rule-based model achieve high scalability and interpretability simultaneously on large datasets?
• RQ2 Can gradient-based optimization be effectively applied to non-differentiable discrete rule models?
• RQ3 Can improved logical activation functions enable end-to-end discretization of continuous features in a differentiable manner?
• RQ4 How does RRL compare in accuracy and interpretability to existing interpretable models across diverse data scales?
• RQ5 Can RRL’s learned rules be meaningfully interpreted and used to guide model understanding and decision-making?

Key findings

• RRL outperforms competitive interpretable models in classification accuracy on both small and large datasets, demonstrating strong generalization.
• Gradient Grafting enables faster and more stable convergence than STE, ProxQuant, and RB on all evaluated datasets, especially where other methods fail to converge.
• The improved logical activation functions significantly enhance scalability, enabling successful training on large datasets like activity, where original functions fail.
• On the sparse facebook dataset, even the original activation functions perform well, indicating robustness in low-density feature spaces.
• The distribution of rule weights in the linear layer reveals that high-weight rules are interpretable and actionable, such as identifying that middle-aged, married clients with low balance are more likely to subscribe to term deposits.
• Model complexity can be controlled via L2 regularization (λ), allowing users to prioritize interpretability (high λ) or accuracy (low λ) based on application needs.