[Paper Review] NeuroRule: A Connectionist Approach to Data Mining

Hongjun Lü, Rudy Setiono+1 more|arXiv (Cornell University)|Jan 5, 2017

Neural Networks and ApplicationsComputer Science28 references107 citations

TL;DR

The paper demonstrates extracting explicit, concise classification rules from neural networks to enable data mining, and compares NeuroRule with decision-tree methods, showing competitive accuracy.

ABSTRACT

Classification, which involves finding rules that partition a given data set into disjoint groups, is one class of data mining problems. Approaches proposed so far for mining classification rules for large databases are mainly decision tree based symbolic learning methods. The connectionist approach based on neural networks has been thought not well suited for data mining. One of the major reasons cited is that knowledge generated by neural networks is not explicitly represented in the form of rules suitable for verification or interpretation by humans. This paper examines this issue. With our newly developed algorithms, rules which are similar to, or more concise than those generated by the symbolic methods can be extracted from the neural networks. The data mining process using neural networks with the emphasis on rule extraction is described. Experimental results and comparison with previously published works are presented.

Motivation & Objective

• Argue for the role of connectionist methods in data mining despite traditional criticisms.
• Develop algorithms to extract explicit, usable classification rules from neural networks.
• Present NeuroRule as a data mining system with training, pruning, and rule-extraction components.
• Empirically evaluate NeuroRule against a decision-tree baseline (C4.5) on benchmark problems.

Proposed method

• Use a three-layer feedforward neural network (input, hidden, output) with a cross-entropy error plus penalty terms for pruning.
• Train with BFGS to expedite convergence.
• Apply a pruning algorithm to remove redundant weights while preserving accuracy.
• Extract explicit rules from the pruned network via a rule-extraction algorithm (RX) that discretizes hidden-node activations and generates rules.
• Optionally create subnetworks for high-degree hidden nodes to simplify rule extraction.
• Compare extracted rules and accuracy against C4.5 on Agrawal et al. benchmark problems.

Experimental results

Research questions

• RQ1 Can explicit, human-interpretable rules be extracted from trained neural networks without sacrificing accuracy?
• RQ2 How does NeuroRule perform in terms of accuracy and rule conciseness compared to symbolically learned methods like C4.5?
• RQ3 What is the impact of network pruning on rule quality and computational efficiency?
• RQ4 Can the rule extraction process scale to large databases with many attributes?

Key findings

• Neural networks can yield concise, extractable rules that often achieve lower error than decision-tree methods on related problems.
• The pruned network for Function 2 used only 17 of 386 initial links, producing compact, interpretable rules.
• Classification accuracy of NeuroRule is comparable to or higher than C4.5 across several benchmark functions (e.g., Functions 1–3, 4–7, 9).
• The rule extraction (RX) process discretizes hidden-node activations to enumerate dependencies and generate rules that map inputs to outputs.
• Rules extracted can closely reproduce the original classification functions, and in simple cases match them exactly.