[Paper Review] Learning Kernel Tests Without Data Splitting

Jonas M. Kübler, Wittawat Jitkrittum+2 more|arXiv (Cornell University)|Jun 1, 2020

Geophysical Methods and Applications6 citations

TL;DR

This paper proposes a selective inference-based method to learn kernel hyperparameters and perform kernel tests on the full dataset without data splitting, enabling higher test power. By calibrating the test threshold in closed form, it maintains valid Type I error control while outperforming data-splitting approaches in empirical power across all split proportions.

ABSTRACT

Modern large-scale kernel-based tests such as maximum mean discrepancy (MMD) and kernelized Stein discrepancy (KSD) optimize kernel hyperparameters on a held-out sample via data splitting to obtain the most powerful test statistics. While data splitting results in a tractable null distribution, it suffers from a reduction in test power due to smaller test sample size. Inspired by the selective inference framework, we propose an approach that enables learning the hyperparameters and testing on the full sample without data splitting. Our approach can correctly calibrate the test in the presence of such dependency, and yield a test threshold in closed form. At the same significance level, our approach's test power is empirically larger than that of the data-splitting approach, regardless of its split proportion.

Motivation & Objective

• To address the power loss in kernel-based hypothesis tests caused by data splitting for hyperparameter tuning.
• To develop a method that enables hyperparameter learning and testing on the full dataset, avoiding sample size reduction.
• To provide a valid, closed-form threshold for the test statistic despite the dependency induced by hyperparameter learning on the same data.
• To improve empirical test power compared to data-splitting approaches across all split ratios.

Proposed method

• Adapt the selective inference framework to kernel tests, treating hyperparameter selection as a model selection event.
• Condition the null distribution on the selected kernel hyperparameters, ensuring valid p-values despite data reuse.
• Derive a closed-form expression for the test threshold that accounts for the selection of hyperparameters.
• Apply the method to maximum mean discrepancy (MMD) and kernelized Stein discrepancy (KSD) tests.
• Use the full dataset for both hyperparameter learning and test statistics computation, eliminating sample splitting.

Experimental results

Research questions

• RQ1 Can we learn kernel hyperparameters and perform kernel tests on the full dataset without compromising Type I error control?
• RQ2 How does the proposed method compare in test power to data-splitting approaches across different split proportions?
• RQ3 Can we derive a closed-form threshold for the test statistic when hyperparameters are learned from the same data used for testing?
• RQ4 Does the method maintain valid inference despite the dependency between hyperparameter selection and test statistics?

Key findings

• The proposed method achieves higher empirical test power than data splitting at the same significance level, regardless of the split proportion.
• The method provides a closed-form threshold for the test statistic, enabling efficient and exact inference.
• Type I error rates are well-controlled, demonstrating the validity of the selective inference calibration.
• The approach eliminates the need for data splitting, thereby preserving all available data for testing.