[Paper Review] DNN2LR: Interpretation-inspired Feature Crossing for Real-world Tabular Data

Zhaocheng Liu, Qiang Liu+2 more|arXiv (Cornell University)|Aug 22, 2020

Video Analysis and Summarization67 references4 citations

TL;DR

DNN2LR proposes a novel method to automatically generate globally interpretable cross features for logistic regression by leveraging interpretation inconsistency in deep neural networks (DNNs). By identifying features with inconsistent local interpretations—caused by hidden-layer interactions—it constructs a compact candidate set of cross features, enabling a simple LR model to outperform complex DNNs and state-of-the-art feature crossing methods on real-world tabular datasets with high efficiency.

ABSTRACT

For sake of reliability, it is necessary for models in real-world applications to be both powerful and globally interpretable. Simple classifiers, e.g., Logistic Regression (LR), are globally interpretable, but not powerful enough to model complex nonlinear interactions among features in tabular data. Meanwhile, Deep Neural Networks (DNNs) have shown great effectiveness for modeling tabular data, but is not globally interpretable. In this work, we find local piece-wise interpretations in DNN of a specific feature are usually inconsistent in different samples, which is caused by feature interactions in the hidden layers. Accordingly, we can design an automatic feature crossing method to find feature interactions in DNN, and use them as cross features in LR. We give definition of the interpretation inconsistency in DNN, based on which a novel feature crossing method called DNN2LR is proposed. Extensive experiments have been conducted on four public datasets and two real-world datasets. The final model, i.e., a LR model empowered with cross features, generated by DNN2LR can outperform the complex DNN model, as well as several state-of-the-art feature crossing methods. The experimental results strongly verify the effectiveness and efficiency of DNN2LR, especially on real-world datasets with large numbers of feature fields.

Motivation & Objective

• To address the trade-off between model performance and global interpretability in real-world tabular data applications.
• To overcome the limitations of Logistic Regression (LR) in capturing complex nonlinear feature interactions without manual feature engineering.
• To develop an efficient, automatic feature crossing method that preserves interpretability while achieving DNN-level performance.
• To identify and exploit interpretation inconsistency in DNNs as a signal for discovering meaningful feature interactions.
• To generate a compact, high-quality candidate set of cross features that enables effective and efficient model training with LR.

Proposed method

• Define interpretation inconsistency in DNNs as the variation in local gradient-based interpretations of a feature across different input samples, which is caused by hidden-layer feature interactions.
• Train a DNN on the full training set to learn complex feature interactions and extract local interpretations via gradient-based saliency maps.
• For each validation sample, compute the interpretation inconsistency score for each feature using the variance of gradients across samples.
• Filter features with high inconsistency (above a quantile threshold η) to identify candidates for feature crossing.
• Construct a global candidate set of second- and higher-order cross feature fields from the filtered features, limited to ε = 3N or 5N to ensure efficiency.
• Train a Logistic Regression model on the candidate set and rank cross features by their contribution on the validation set to select the final set of useful cross features.

Experimental results

Research questions

• RQ1 Can interpretation inconsistency in DNNs be used as a reliable signal to identify meaningful feature interactions for cross feature generation?
• RQ2 How does DNN2LR compare in performance and efficiency to state-of-the-art feature crossing methods like AutoCross and AutoFM on real-world datasets?
• RQ3 What is the impact of hyper-parameters such as the quantile threshold η and candidate set size ε on DNN2LR’s performance and robustness?
• RQ4 Can a simple LR model enhanced with DNN2LR-generated cross features outperform a complex DNN model in terms of accuracy and generalization?
• RQ5 How scalable is DNN2LR on datasets with a large number of feature fields, especially in real-world applications?

Key findings

• DNN2LR-generated cross features enable a simple Logistic Regression model to outperform complex DNN models on all four public and two real-world datasets, demonstrating superior generalization and performance.
• On real-world datasets with large numbers of feature fields, DNN2LR reduces running time by 6× to 50× compared to AutoCross and AutoFM, highlighting its high efficiency.
• The method achieves stable performance across different hyper-parameter settings, especially with η = 5% and ε = 3N (or 5N for small datasets), indicating robustness and low sensitivity.
• The final model uses only 3N to 5N candidate cross features, a dramatically smaller set than the full combinatorial space, proving the method’s efficiency and scalability.
• Second-order cross features are more prevalent than higher-order ones, but higher-order features still contribute to performance gains, suggesting a hierarchy of interaction importance.
• On the Employee dataset (only 9 features), DNN2LR with ε = 5N shows a relative performance improvement, indicating adaptability even in low-dimensional settings.