[Paper Review] Estimation and inference for transfer learning with high-dimensional quantile regression

Jiayu Huang, Mingqiu Wang+1 more|arXiv (Cornell University)|Nov 26, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a high-dimensional quantile regression framework for transfer learning that accommodates heterogeneity and heavy-tailed distributions in source and target domains. By introducing a double transfer learning estimator and data-splitting-based transferability detection, it achieves improved estimation accuracy, valid inference via confidence intervals, and robustness against negative transfer, with theoretical error bounds and empirical validation showing strong performance.

ABSTRACT

Transfer learning has become an essential technique to exploit information from the source domain to boost performance of the target task. Despite the prevalence in high-dimensional data, heterogeneity and heavy tails are insufficiently accounted for by current transfer learning approaches and thus may undermine the resulting performance. We propose a transfer learning procedure in the framework of high-dimensional quantile regression models to accommodate heterogeneity and heavy tails in the source and target domains. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains, showing that lower error bounds can be achieved for critical selection criterion and larger sample size of source tasks. We further propose valid confidence interval and hypothesis test procedures for individual component of high-dimensional quantile regression coefficients by advocating a double transfer learning estimator, which is one-step debiased estimator for the transfer learning estimator wherein the technique of transfer learning is designed again. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer and identify transferable sources with high probability. Simulation results demonstrate that the proposed method exhibits some favorable and compelling performances and the practical utility is further illustrated by analyzing a real example.

Motivation & Objective

• To address the limitations of existing transfer learning methods in handling high-dimensional, heteroscedastic, and heavy-tailed data.
• To develop a theoretically grounded transfer learning procedure within high-dimensional quantile regression that ensures estimation accuracy and statistical inference.
• To provide valid confidence intervals and hypothesis tests for individual regression coefficients in high-dimensional settings.
• To detect transferable source domains with high probability and avoid negative transfer using a data-splitting strategy.
• To establish theoretical error bounds for the transfer learning estimator under controlled selection of source domains.

Proposed method

• Proposes a double transfer learning estimator: a one-step debiased version of the transfer learning estimator to enable valid inference.
• Introduces a data-splitting technique to construct a transferability detection procedure that identifies reliable source domains with high probability.
• Employs a carefully designed selection criterion for source domains based on their relevance and sample size to minimize estimation error.
• Uses high-dimensional quantile regression models to model conditional quantiles, accommodating heterogeneity and heavy-tailed error distributions.
• Derives theoretical error bounds for the transfer learning estimator under assumptions on sparsity and design matrix properties.
• Applies a debiasing mechanism to the transfer learning estimator to correct for estimation bias and enable asymptotically valid inference.

Experimental results

Research questions

• RQ1 Can transfer learning in high-dimensional quantile regression effectively handle heavy-tailed and heterogeneous data across domains?
• RQ2 How can valid confidence intervals and hypothesis tests be constructed for high-dimensional quantile regression coefficients under transfer learning?
• RQ3 What criteria ensure that the selection of source domains leads to improved estimation accuracy and avoids negative transfer?
• RQ4 To what extent does the data-splitting-based transferability detection procedure reliably identify useful source domains?
• RQ5 What theoretical error bounds can be established for the transfer learning estimator under controlled source domain selection?

Key findings

• The proposed transfer learning estimator achieves lower error bounds when the selection criterion for source domains is critical and when source sample sizes are large.
• The double transfer learning estimator enables valid confidence intervals and hypothesis tests for individual regression coefficients with asymptotic coverage guarantees.
• The data-splitting-based transferability detection method ensures high-probability identification of transferable sources and prevents negative transfer.
• Simulation results demonstrate that the method outperforms baseline approaches in estimation accuracy and inference reliability under heavy-tailed and heterogeneous conditions.
• The real data example illustrates the practical utility of the method in a high-dimensional setting with complex error structures.
• Theoretical analysis confirms that the error bounds scale favorably with the number of transferable source samples and the strength of the selection criterion.