[Paper Review] Adaptive Non-Parametric Regression With the $K$-NN Fused Lasso

Oscar Hernán Madrid Padilla, James Sharpnack+2 more|arXiv (Cornell University)|Jul 31, 2018

Statistical Methods and Inference57 references3 citations

TL;DR

This paper introduces the K-NN fused lasso, a non-parametric regression method that combines K-nearest neighbors (K-NN) graph construction with fused lasso regularization to enable local and manifold adaptivity. It achieves nearly minimax optimal rates in mean squared error for piecewise Lipschitz and bounded variation functions on irregularly spaced design points.

ABSTRACT

The fused lasso, also known as total-variation denoising, is a locally-adaptive function estimator over a regular grid of design points. In this paper, we extend the fused lasso to settings in which the points do not occur on a regular grid, leading to an approach for non-parametric regression. This approach, which we call the $K$-nearest neighbors ($K$-NN) fused lasso, involves (i) computing the $K$-NN graph of the design points; and (ii) performing the fused lasso over this $K$-NN graph. We show that this procedure has a number of theoretical advantages over competing approaches: specifically, it inherits local adaptivity from its connection to the fused lasso, and it inherits manifold adaptivity from its connection to the $K$-NN approach. We show that excellent results are obtained in a simulation study and on an application to flu data. For completeness, we also study an estimator that makes use of an $ε$-graph rather than a $K$-NN graph, and contrast this with the $K$-NN fused lasso.

Motivation & Objective

• To extend the fused lasso—originally designed for regular grids—to irregularly spaced design points in non-parametric regression.
• To endow the fused lasso with local adaptivity by leveraging the K-NN graph structure to capture local density variations.
• To achieve manifold adaptivity by allowing the method to adapt to the intrinsic dimensionality of the data manifold.
• To provide theoretical guarantees on estimation error and consistency under general design point configurations.
• To compare the K-NN fused lasso with an alternative ε-graph-based estimator and evaluate performance on real and simulated data.

Proposed method

• Construct a K-NN graph over the design points $x_1, dots, x_n$ using the metric $d_{ ext{X}}$, where each point connects to its K nearest neighbors.
• Apply fused lasso regularization on the K-NN graph by solving the optimization problem $\min_{\theta \in \mathbb{R}^n} \left\{ \frac{1}{2} \sum_{i=1}^n (y_i - \theta_i)^2 + \lambda \sum_{(i,j) \in E} |\theta_i - \theta_j| \right\}$, where $E$ is the edge set of the K-NN graph.
• Use the resulting estimator $\hat{\theta}_i$ as the predicted value $\hat{f}(x_i)$ at each design point.
• Theoretical analysis relies on controlling graph degrees and mesh counts via probabilistic concentration bounds under regularity assumptions on the design density and function smoothness.
• Establish consistency and nearly minimax optimal rates by leveraging connections to total variation denoising and graph-based regularization.
• Compare with an ε-graph variant, where edges are formed between points within distance ε, and analyze its theoretical performance under similar assumptions.

Experimental results

Research questions

• RQ1 Can the fused lasso be extended to irregularly spaced design points while preserving its local adaptivity and theoretical optimality?
• RQ2 Does combining the fused lasso with K-NN graph construction yield a method that adapts to the intrinsic dimensionality of the data manifold?
• RQ3 What are the theoretical error bounds and consistency properties of the K-NN fused lasso under general design configurations?
• RQ4 How does the K-NN fused lasso compare to an ε-graph-based alternative in terms of estimation accuracy and theoretical guarantees?
• RQ5 Can the method achieve nearly minimax optimal rates in mean squared error for piecewise Lipschitz and bounded variation functions on irregular grids?

Key findings

• The K-NN fused lasso inherits local adaptivity from the fused lasso, enabling better estimation in regions of high design point density.
• The method achieves manifold adaptivity by automatically adjusting resolution based on local point density, improving performance in non-uniformly sampled data.
• Theoretical analysis shows that the K-NN fused lasso estimator achieves a mean squared error bound of $\mathbb{E}\left[\sum_{i=1}^n (\theta_i^* - \hat{\theta}_{K,i})^2 \mid \Lambda\right] \geq C_1 n^{3/4 - a}$ for some $C_1 > 0$ and small $a > 0$, under regularity conditions.
• The method is consistent for denoising when the total variation of the signal along the K-NN graph divided by $n$ tends to zero.
• Analogous theoretical guarantees are established for the ε-graph variant, with similar error bounds under comparable assumptions.
• Simulation studies and an application to flu data demonstrate strong empirical performance, with improved estimation accuracy over competing non-parametric methods.