[Paper Review] Interpretable Low-Dimensional Regression via Data-Adaptive Smoothing

Wesley Tansey, Jesse Thomason+1 more|arXiv (Cornell University)|Aug 6, 2017

Statistical Methods and Inference12 references3 citations

TL;DR

This paper proposes Maximum Variance Total Variation (MVTV) denoising, a data-adaptive method for interpretable low-dimensional regression that partitions feature space into blocks of constant value using a maximum-variance heuristic and applies total variation denoising for smoothing. MVTV achieves superior accuracy and interpretability over CART and CRISP, as validated by both AIC-based metrics and a human study showing statistically significant improvement in predictive interpretability.

ABSTRACT

We consider the problem of estimating a regression function in the common situation where the number of features is small, where interpretability of the model is a high priority, and where simple linear or additive models fail to provide adequate performance. To address this problem, we present Maximum Variance Total Variation denoising (MVTV), an approach that is conceptually related both to CART and to the more recent CRISP algorithm, a state-of-the-art alternative method for interpretable nonlinear regression. MVTV divides the feature space into blocks of constant value and fits the value of all blocks jointly via a convex optimization routine. Our method is fully data-adaptive, in that it incorporates highly robust routines for tuning all hyperparameters automatically. We compare our approach against CART and CRISP via both a complexity-accuracy tradeoff metric and a human study, demonstrating that that MVTV is a more powerful and interpretable method.

Motivation & Objective

• To address the challenge of achieving high interpretability in low-dimensional regression when linear or additive models fail to capture complex nonlinear relationships.
• To develop a method that balances model accuracy and interpretability by creating sharp, data-adaptive partitions of the feature space into constant-value blocks.
• To automate hyperparameter tuning (grid size and regularization) using data-adaptive, robust procedures to eliminate manual intervention.
• To evaluate the interpretability of the resulting model not only through statistical metrics but also via human perception studies.
• To demonstrate that MVTV outperforms CART and CRISP in both predictive accuracy and human interpretability on real-world data, such as Austin crime data.

Proposed method

• MVTV uses a maximum-variance heuristic to create a data-adaptive grid over the feature space, ensuring that grid boundaries align with significant changes in the underlying signal.
• The method partitions the feature space into blocks where each block is assigned a constant predicted value, enabling interpretable, piecewise-constant regression.
• It applies fast total variation denoising over the resulting grid to smooth predictions while preserving sharp transitions at boundaries, using a convex optimization framework.
• The approach jointly optimizes block values and grid structure via a convex problem, with regularization via group fused lasso to encourage spatial smoothness.
• Hyperparameters (grid size $q$ and regularization $ heta$) are selected automatically using cross-validation and a solution path approach, ensuring robustness and no manual tuning.
• The method leverages a sparse binary selector matrix to map input features to block predictions and uses ADMM for efficient optimization, with complexity $ ilde{O}(n + q^4)$ for initial setup and $ ilde{O}(q^3)$ per iteration.

Experimental results

Research questions

• RQ1 Can a data-adaptive, interpretable regression method achieve better accuracy and interpretability than CART and CRISP in low-dimensional settings?
• RQ2 Does a maximum-variance heuristic for grid construction lead to more meaningful and informative partitions of the feature space compared to uniform or quantile-based grids?
• RQ3 Can automatic hyperparameter tuning via cross-validation improve model performance without requiring manual intervention?
• RQ4 To what extent does MVTV enhance human interpretability in predictive tasks compared to raw data, CART, or CRISP?
• RQ5 Is there a significant improvement in human prediction accuracy when using MVTV-processed data over other smoothing methods in a controlled study?

Key findings

• MVTV achieved the lowest AIC score (10327.58) on the Austin crime dataset, outperforming CART (11139.29) and CRISP (18326.33), indicating a better balance between model fit and complexity.
• In a human study with 207 annotators, MVTV reduced human prediction error to 2.75 × 10⁻², significantly lower than CART (3.24 × 10⁻²) and CRISP (3.99 × 10⁻²), and statistically outperformed raw data.
• MVTV was the only method to achieve a statistically significant improvement in human prediction accuracy over raw data, as confirmed by Tukey’s range test.
• CART over-smoothed the data, creating large uniform regions that misled annotators into predicting the uniform value, resulting in high error despite low variance.
• CRISP under-smoothed the data, producing noisy patches that made prediction difficult, leading to higher human error than MVTV.
• The MVTV method produced visually interpretable, flexible partitions that balanced local detail and global structure, effectively capturing spatial patterns in the Austin crime data.