[Paper Review] Doubly Robust Augmented Model Accuracy Transfer Inference with High Dimensional Features

Doudou Zhou, Molei Liu+2 more|arXiv (Cornell University)|Aug 10, 2022

Cancer-related molecular mechanisms research4 citations

TL;DR

This paper proposes DRAMATIC, a doubly robust method for transferring and evaluating model accuracy in high-dimensional, unlabeled target populations using labeled source data under covariate shift. By combining imputation and density ratio models with calibrated moment equations, DRAMATIC achieves $n^{1/2}$-consistent estimation of ROC curves and performance metrics, outperforming existing methods in bias reduction and coverage under distributional shifts.

ABSTRACT

Due to label scarcity and covariate shift happening frequently in real-world studies, transfer learning has become an essential technique to train models generalizable to some target populations using existing labeled source data. Most existing transfer learning research has been focused on model estimation, while there is a paucity of literature on transfer inference for model accuracy despite its importance. We propose a novel $\mathbf{D}$oubly $\mathbf{R}$obust $\mathbf{A}$ugmented $\mathbf{M}$odel $\mathbf{A}$ccuracy $\mathbf{T}$ransfer $\mathbf{I}$nferen$\mathbf{C}$e (DRAMATIC) method for point and interval estimation of commonly used classification performance measures in an unlabeled target population using labeled source data. Specifically, DRAMATIC derives and evaluates the risk model for a binary response $Y$ against some low dimensional predictors $\mathbf{A}$ on the target population, leveraging $Y$ from source data only and high dimensional adjustment features $\mathbf{X}$ from both the source and target data. The proposed estimators are doubly robust in the sense that they are $n^{1/2}$ consistent when at least one model is correctly specified and certain model sparsity assumptions hold. Simulation results demonstrate that the point estimation have negligible bias and the confidence intervals derived by DRAMATIC attain satisfactory empirical coverage levels. We further illustrate the utility of our method to transfer the genetic risk prediction model and its accuracy evaluation for type II diabetes across two patient cohorts in Mass General Brigham (MGB) collected using different sampling mechanisms and at different time points.

Motivation & Objective

• Address the critical gap in transfer learning for model performance evaluation, particularly when gold-standard labels are scarce in target populations.
• Enable accurate point and interval estimation of classification performance metrics—such as AUC and ROC—on an unlabeled target population using labeled source data.
• Overcome the challenges of high-dimensional covariates and distributional shifts (covariate shift) between source and target populations.
• Develop a method that remains consistent and efficient even when one of the two key models (imputation or density ratio) is misspecified, ensuring robustness.
• Mitigate regularization bias from high-dimensional nuisance estimators in ROC curve estimation through a novel calibrated moment equation framework.

Proposed method

• Formulate a doubly robust estimating equation for the risk model $\mathbb{P}(Y=1 \mid \mathbf{A})$ on the target population using only source data on $Y$ and high-dimensional features $\mathbf{X}$ from both source and target.
• Simultaneously model the imputation of $\mathbb{E}[Y \mid \mathbf{X}]$ and the density ratio $\frac{p_{\text{target}}(\mathbf{X})}{p_{\text{source}}(\mathbf{X})}$ to account for covariate shift.
• Construct calibrated moment equations to correct for regularization bias in high-dimensional nuisance estimators of the imputation and density ratio models.
• Apply a novel 'nearest quantile' strategy to ensure statistical efficiency and computational feasibility in estimating the entire ROC curve.
• Derive doubly robust estimators for AUC, ROC(0.1), and ROC(0.2) with valid 95% confidence intervals using the calibrated moment framework.
• Use sparsity assumptions on $l_r$ norms (for $r \in [0,1]$) of nuisance parameters rather than strict $l_0$ sparsity, enhancing model flexibility and realism.

Figure 1 : The nearest quantile strategy. The quantiles $\widehat{c}_{j}$ ’s are set to divide the source sample equally such that $n_{j}-n_{j+1}=n_{\min}$ and $n_{m}=n_{\min}$ . The nuisance models are calibrated at these quantiles for the doubly robust estimation of $\mathbf{TPR}$ and $\mathbf{FPR

Experimental results

Research questions

• RQ1 Can we achieve consistent and efficient estimation of model accuracy metrics (e.g., AUC, ROC) in an unlabeled target population when only labeled source data are available?
• RQ2 How can we ensure robustness in model accuracy transfer when either the imputation model or the density ratio model is misspecified?
• RQ3 What is an effective way to correct for regularization bias in high-dimensional nuisance estimators when estimating the full ROC curve?
• RQ4 Can we construct valid confidence intervals for performance metrics under distributional shift with high-dimensional features?
• RQ5 How does the proposed method compare to existing approaches (e.g., importance weighting, imputation, naive source estimation) in terms of bias, coverage, and accuracy?

Key findings

• DRAMATIC achieved the smallest total variation (TV) distance to the validation benchmark ROC curve—41% smaller than the naive source estimator, 58% smaller than importance weighting, and 32% smaller than imputation-based methods.
• The rMSPE (relative mean squared prediction error) of DRAMATIC was 68% lower than the naive source estimator, 54% lower than importance weighting, and 34% lower than the imputation method.
• The 95% confidence intervals for AUC and ROC(0.1) produced by DRAMATIC correctly covered the validation estimator and had moderate, reasonable lengths.
• In terms of AUC, DRAMATIC’s estimate (0.671) was closest to the target (0.656), with a 95% CI of (0.622, 0.720), outperforming all other methods in proximity to the true value.
• The estimated ROC curve from DRAMATIC (red dashed line in Figure 2) was visually and quantitatively closest to the validation benchmark (black solid line), confirming superior curve estimation.
• The method demonstrated robustness under model misspecification: consistent estimation was achieved when either the imputation or density ratio model was correctly specified, under appropriate sparsity conditions.

Figure 2 : Estimated ROC curves. Source: naive source data estimator; DRAMATIC: our proposed doubly robust method; IW: importance weighting method; IM: imputation based method; Target: benchmark obtained using the validation samples.