[Paper Review] Transfer Learning with Large-Scale Quantile Regression

Jun Jin, Jun Yan+2 more|arXiv (Cornell University)|Dec 13, 2022

Advanced Statistical Methods and Models4 citations

TL;DR

This paper proposes a novel transfer learning framework for high-dimensional quantile regression that identifies and leverages informative external data sources to improve estimation accuracy for a target population. By using sample splitting to detect sources with similar models to the target, the method achieves significantly lower error rates than naive estimators, especially under low sample sizes and high signal-to-noise ratios.

ABSTRACT

Quantile regression is increasingly encountered in modern big data applications due to its robustness and flexibility. We consider the scenario of learning the conditional quantiles of a specific target population when the available data may go beyond the target and be supplemented from other sources that possibly share similarities with the target. A crucial question is how to properly distinguish and utilize useful information from other sources to improve the quantile estimation and inference at the target. We develop transfer learning methods for high-dimensional quantile regression by detecting informative sources whose models are similar to the target and utilizing them to improve the target model. We show that under reasonable conditions, the detection of the informative sources based on sample splitting is consistent. Compared to the naive estimator with only the target data, the transfer learning estimator achieves a much lower error rate as a function of the sample sizes, the signal-to-noise ratios, and the similarity measures among the target and the source models. Extensive simulation studies demonstrate the superiority of our proposed approach. We apply our methods to tackle the problem of detecting hard-landing risk for flight safety and show the benefits and insights gained from transfer learning of three different types of airplanes: Boeing 737, Airbus A320, and Airbus A380.

Motivation & Objective

• Address the challenge of estimating conditional quantiles in high-dimensional, small-sample settings where target data is limited.
• Overcome negative transfer by distinguishing between relevant and irrelevant external data sources in transfer learning.
• Develop a consistent method to detect informative sources whose models are similar to the target model.
• Improve estimation accuracy and inference for the target population by leveraging heterogeneous external data while accounting for feature distribution, model, and error distribution differences.
• Demonstrate the method's superiority through simulations and real-world application to hard-landing risk detection in commercial aviation.

Proposed method

• Use sample splitting to construct test statistics for detecting sources whose models are similar to the target model.
• Apply a two-stage procedure: first detect informative sources via hypothesis testing on model similarity, then combine them with target data in a fused quantile regression model.
• Employ Lasso-type regularization in the quantile regression framework to handle high-dimensional feature spaces.
• Formulate a transfer learning estimator that integrates target data and selected informative sources, improving estimation efficiency.
• Ensure consistency of source detection under reasonable regularity conditions, including sparsity and sub-Gaussian error assumptions.
• Use cross-validation and model selection criteria to tune regularization parameters and optimize performance.

Experimental results

Research questions

• RQ1 Can we consistently detect external data sources whose models are similar to the target model in high-dimensional quantile regression?
• RQ2 How does the proposed transfer learning method compare to naive estimators that use only target data or combine all data indiscriminately?
• RQ3 To what extent does the error rate of the transfer learning estimator depend on sample size, signal-to-noise ratio, and model similarity?
• RQ4 Can transfer learning improve estimation accuracy in real-world applications with limited target data, such as hard-landing risk prediction for rare aircraft types?
• RQ5 How does the method handle heterogeneity in feature distributions, model structures, and error distributions across sources and the target?

Key findings

• The proposed method achieves a significantly lower mean squared error (MSE) than the naive estimator that uses only target data, especially when the target sample size is small.
• Source detection via sample splitting is consistent under regularity conditions, meaning the method correctly identifies informative sources with high probability as sample sizes grow.
• The transfer learning estimator reduces error rates more substantially when signal-to-noise ratios are high and source-target model similarity is strong.
• In the flight safety application, the method successfully improved hard-landing risk prediction for the Airbus A380 using data from Boeing 737 and Airbus A320, which are similar but not identical in operational characteristics.
• The method demonstrated robustness to non-normal error distributions and heteroscedasticity, as evidenced by Q-Q plots and residual analysis in the QAR data application.
• The inclusion of multiple diverse but similar sources (e.g., different aircraft types) led to better generalization and more stable quantile estimates than using a single source or all sources indiscriminately.