[Paper Review] Quantile Regression Under Memory Constraint

Xi Chen, Weidong Liu+1 more|RePEc: Research Papers in Economics|Oct 18, 2018

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a computationally efficient quantile regression method under memory constraints, where only a small batch of data (size m) can be stored. It uses iterative refinement via multiple aggregation rounds starting from an initial estimator on a small batch, achieving asymptotic normality and full efficiency with only a few rounds, even as sample size n grows polynomially in m and dimension p increases.

ABSTRACT

This paper studies the inference problem in quantile regression (QR) for a large sample size $n$ but under a limited memory constraint, where the memory can only store a small batch of data of size $m$. A natural method is the naïve divide-and-conquer approach, which splits data into batches of size $m$, computes the local QR estimator for each batch, and then aggregates the estimators via averaging. However, this method only works when $n=o(m^2)$ and is computationally expensive. This paper proposes a computationally efficient method, which only requires an initial QR estimator on a small batch of data and then successively refines the estimator via multiple rounds of aggregations. Theoretically, as long as $n$ grows polynomially in $m$, we establish the asymptotic normality for the obtained estimator and show that our estimator with only a few rounds of aggregations achieves the same efficiency as the QR estimator computed on all the data. Moreover, our result allows the case that the dimensionality $p$ goes to infinity. The proposed method can also be applied to address the QR problem under distributed computing environment (e.g., in a large-scale sensor network) or for real-time streaming data.

Motivation & Objective

• Address the challenge of performing quantile regression on large datasets when memory can only store a small batch of size m.
• Overcome the limitations of the naïve divide-and-conquer approach, which is inefficient and only valid when n=o(m²).
• Develop a method that maintains statistical efficiency while operating under strict memory constraints.
• Enable inference in high-dimensional settings where p can grow to infinity.
• Support applications in distributed computing and real-time streaming data environments.

Proposed method

• Propose an iterative aggregation framework that starts with a single small batch estimator and refines it over multiple rounds.
• Use successive rounds of aggregation to update the estimator, leveraging local QR estimates from batches and combining them via weighted averaging.
• Introduce a recursive refinement mechanism that improves estimator accuracy without storing all data.
• The method relies on asymptotic normality of the initial estimator and propagates efficiency through iterative corrections.
• Theoretical analysis shows that the final estimator achieves the same asymptotic variance as the full-data QR estimator.
• The approach is robust to heteroscedastic and heavy-tailed errors, as validated in simulations.

Experimental results

Research questions

• RQ1 Can a quantile regression estimator be efficiently computed under strict memory constraints where only m data points can be stored?
• RQ2 Does an iterative aggregation method outperform naïve divide-and-conquer in terms of computational cost and statistical efficiency?
• RQ3 Can the proposed method achieve the same asymptotic efficiency as the full-data QR estimator when n grows polynomially in m?
• RQ4 How does the method perform in high-dimensional settings where p→∞?
• RQ5 Can the method be extended to distributed and streaming data environments without loss of efficiency?

Key findings

• The proposed method achieves asymptotic normality of the estimator under the condition that n grows polynomially in m, even when p→∞.
• With only a few rounds of aggregation (e.g., q=4 or 8), the method attains the same asymptotic variance as the full-data QR estimator.
• In simulations, the coverage rates of confidence intervals are close to the nominal 95% level, with bias and variance decreasing significantly with more rounds.
• For n=2×10⁶ and m=20,000, the DC LEQR with q=8 achieves a coverage rate of 0.955 and variance of 12.46×10⁻⁴, approaching the full-data QR performance.
• Computation time is significantly reduced compared to ADMM-based QR: for n=10⁷ and m=1000, the method takes 9.344s vs. 11.423s for ADMM, with better coverage and lower bias.
• The method outperforms the naïve divide-and-conquer approach, which shows poor coverage (e.g., 0.739–0.808) and high bias (up to 5.40×10⁻²), especially in large n settings.