[Paper Review] Computation of projection regression depth and its induced median

Yijun Zuo|arXiv (Cornell University)|May 28, 2019

Advanced Statistical Methods and Models33 references4 citations

TL;DR

This paper presents exact and approximate algorithms for computing projection regression depth (PRD) and its induced median (maximum depth estimator), achieving O(n² log n) and O(RpNβ(p² + nNvNIter)) computational costs respectively. It introduces three fast, robust PRD-based estimators that are up to 30 times faster than the PRD median while maintaining high accuracy and outperforming the leading regression depth median in robustness and efficiency under contamination.

ABSTRACT

Notions of depth in regression have been introduced and studied in the literature. The most famous example is Regression Depth (RD), which is a direct extension of location depth to regression. The projection regression depth (PRD) is the extension of another prevailing location depth, the projection depth, to regression. The computation issues of the RD have been discussed in the literature. The computation issues of the PRD have never been dealt with before. The computation issues of the PRD and its induced median (maximum depth estimator) in a regression setting are addressed now. For a given $\bsβ\in\R^p$ exact algorithms for the PRD with cost $O(n^2\log n)$ ($p=2$) and $O(N(n, p)(p^{3}+n\log n+np^{1.5}+npN_{Iter}))$ ($p>2$) and approximate algorithms for the PRD and its induced median with cost respectively $O(N_{\mb{v}}np)$ and $O(Rp N_{\bsβ}(p^2+nN_{\mb{v}}N_{Iter}))$ are proposed. Here $N(n, p)$ is a number defined based on the total number of $(p-1)$ dimensional hyperplanes formed by points induced from sample points and the $\bsβ$; $N_{\mb{v}}$ is the total number of unit directions $\mb{v}$ utilized; $N_{\bsβ}$ is the total number of candidate regression parameters $\bsβ$ employed; $N_{Iter}$ is the total number of iterations carried out in an optimization algorithm; $R$ is the total number of replications. Furthermore, as the second major contribution, three PRD induced estimators, which can be computed up to 30 times faster than that of the PRD induced median while maintaining a similar level of accuracy are introduced. Examples and simulation studies reveal that the depth median induced from the PRD is favorable in terms of robustness and efficiency, compared to the maximum depth estimator induced from the RD, which is the current leading regression median.

Motivation & Objective

• To address the long-standing gap in computational methods for projection regression depth (PRD), a robust regression depth notion extending projection depth to regression.
• To develop exact and approximate algorithms for PRD computation and its induced median, enabling practical application in high-dimensional regression.
• To introduce three new PRD-induced estimators that are computationally efficient while preserving robustness and accuracy.
• To compare the PRD median with the established regression depth (RD) median in terms of robustness, efficiency, and breakdown point under contamination.
• To demonstrate through simulations that the PRD median offers superior performance in finite samples, especially under contamination.

Proposed method

• Proposes an exact algorithm for PRD computation with O(n² log n) cost for p=2 and O(N(n,p)(p³ + n log n + np¹·⁵ + npNIter)) cost for p>2, based on hyperplane enumeration and optimization.
• Introduces an approximate algorithm with O(Nvnp) cost for PRD and O(RpNβ(p² + nNvNIter)) cost for the induced median, using random unit directions and iterative optimization.
• Defines unfitness (UF) as the supremum of absolute correlation between projected residuals and unit vectors, forming the basis of PRD computation.
• Employs a projection-pursuit framework to compute depth by maximizing robustness across all directions, ensuring affine invariance and scale invariance.
• Develops three fast estimators by leveraging directional search and iterative refinement, reducing computational burden without sacrificing robustness.
• Uses Monte Carlo simulations and real-data examples to validate the algorithms and compare performance with existing methods like least squares and RD median.

Experimental results

Research questions

• RQ1 How can projection regression depth (PRD) be computed efficiently in high-dimensional regression settings?
• RQ2 What are the computational complexities of exact and approximate algorithms for PRD and its induced median?
• RQ3 Can faster estimators be derived from PRD that maintain high robustness and accuracy while reducing computation time?
• RQ4 How does the PRD-induced median compare to the regression depth (RD)-induced median in terms of robustness and efficiency under contamination?
• RQ5 What is the finite-sample performance of PRD-based estimators in terms of mean squared error and breakdown point?

Key findings

• The PRD-induced median (T*PRD) has a finite-sample breakdown point of up to 50%, significantly higher than the 33% of the RD-induced median (T*RD).
• The PRD-induced median is uniformly more efficient than the RD-induced median across all sample sizes n, with efficiency increasing to 525% at n=200 under contamination.
• The PRD-induced median outperforms the least squares estimator in contaminated settings, remaining stable even when one or three points are severely contaminated.
• The three proposed fast estimators achieve up to 30 times faster computation than the PRD median while maintaining similar accuracy and robustness.
• The PRD median is unique and less sensitive to contamination than the RD median, which can suffer from non-uniqueness and low finite-sample breakdown points.
• Simulations show that the EMSE of all depth-based estimators decreases with increasing n, and the PRD-based estimators consistently achieve lower EMSE than least squares and RD-based estimators under contamination.