[Paper Review] Nonregular and Minimax Estimation of Individualized Thresholds in High Dimension with Binary Responses

Huijie Feng, Yang Ning+1 more|arXiv (Cornell University)|May 26, 2019

Statistical Methods and Inference37 references4 citations

TL;DR

This paper proposes a regularized smoothed loss approach for high-dimensional estimation of individualized linear thresholds in binary response models, addressing nonregularity and computational intractability. It establishes a nonstandard $$(s\log d/n)^{\beta/(2\beta+1)}$ error rate, proven minimax optimal up to a logarithmic factor, with adaptive Lepski’s method and a path-following algorithm ensuring geometric convergence.

ABSTRACT

Given a large number of covariates $Z$, we consider the estimation of a high-dimensional parameter $θ$ in an individualized linear threshold $θ^T Z$ for a continuous variable $X$, which minimizes the disagreement between $ ext{sign}(X-θ^TZ)$ and a binary response $Y$. While the problem can be formulated into the M-estimation framework, minimizing the corresponding empirical risk function is computationally intractable due to discontinuity of the sign function. Moreover, estimating $θ$ even in the fixed-dimensional setting is known as a nonregular problem leading to nonstandard asymptotic theory. To tackle the computational and theoretical challenges in the estimation of the high-dimensional parameter $θ$, we propose an empirical risk minimization approach based on a regularized smoothed loss function. The statistical and computational trade-off of the algorithm is investigated. Statistically, we show that the finite sample error bound for estimating $θ$ in $\ell_2$ norm is $(s\log d/n)^{β/(2β+1)}$, where $d$ is the dimension of $θ$, $s$ is the sparsity level, $n$ is the sample size and $β$ is the smoothness of the conditional density of $X$ given the response $Y$ and the covariates $Z$. The convergence rate is nonstandard and slower than that in the classical Lasso problems. Furthermore, we prove that the resulting estimator is minimax rate optimal up to a logarithmic factor. The Lepski's method is developed to achieve the adaption to the unknown sparsity $s$ and smoothness $β$. Computationally, an efficient path-following algorithm is proposed to compute the solution path. We show that this algorithm achieves geometric rate of convergence for computing the whole path. Finally, we evaluate the finite sample performance of the proposed estimator in simulation studies and a real data analysis.

Motivation & Objective

• To address the computational intractability and nonregular asymptotic behavior in high-dimensional estimation of individualized thresholds for binary responses.
• To develop a statistically optimal and computationally efficient method for estimating a high-dimensional parameter $\bm{\theta}$ in a linear threshold model $\bm{\theta}^T\bm{Z}$.
• To establish finite-sample error bounds and minimax optimality for the estimator under unknown sparsity $s$ and smoothness $\beta$.
• To enable adaptation to unknown $s$ and $\beta$ via Lepski’s method and ensure geometric convergence in computation.

Proposed method

• Formulates the threshold estimation as an M-estimation problem minimizing empirical risk with a discontinuous sign function.
• Introduces a regularized smoothed loss function to replace the non-differentiable sign function, enabling tractable optimization.
• Establishes Fisher consistency as the smoothing bandwidth $\delta \to 0$, ensuring the method converges to the true risk minimizer.
• Derives an $\ell_2$ error bound of $(s\log d/n)^{\beta/(2\beta+1)}$ under smoothness $\beta$ of the conditional density of $X$ given $Y$ and $\bm{Z}$.
• Applies Lepski’s method to adaptively select tuning parameters without prior knowledge of $s$ and $\beta$.
• Develops a path-following algorithm with geometric rate of convergence for computing the full solution path efficiently.

Experimental results

Research questions

• RQ1 Can a computationally feasible and statistically optimal method be developed for high-dimensional threshold estimation with binary responses under nonregularity?
• RQ2 What is the optimal convergence rate for estimating $\bm{\theta}$ in the $\ell_2$ norm under unknown smoothness $\beta$ and sparsity $s$?
• RQ3 How can adaptation to unknown $s$ and $\beta$ be achieved in high-dimensional threshold estimation?
• RQ4 Can a non-convex, non-smooth empirical risk minimization problem be effectively solved via a regularized smoothed loss approach?
• RQ5 Does the proposed method achieve minimax optimality up to a logarithmic factor in the high-dimensional, nonregular setting?

Key findings

• The proposed estimator achieves an $\ell_2$ error bound of $(s\log d/n)^{\beta/(2\beta+1)}$, which is nonstandard and slower than classical Lasso rates.
• The convergence rate is minimax optimal up to a logarithmic factor, establishing theoretical optimality of the method.
• Lepski’s method enables adaptation to unknown sparsity $s$ and smoothness $\beta$ without requiring prior knowledge of these parameters.
• The path-following algorithm achieves geometric rate of convergence, ensuring efficient computation of the full solution path.
• Simulation studies and real data analysis from the ChAMP trial confirm the method's finite-sample performance and robustness across different variable selection patterns.
• The method outperforms alternatives like SVM and logistic regression in variable selection, with clinically relevant variables like KSymp_3mo and SF36Soc_6mo consistently selected with negative coefficients.