[Paper Review] Learning Deep Kernels for Non-Parametric Two-Sample Tests

Feng Liu, Wenkai Xu+4 more|arXiv (Cornell University)|Feb 21, 2020

Machine Learning and Data Classification66 citations

TL;DR

The paper introduces deep-kernel based two-sample tests that learn kernels via deep nets to maximize test power, outperforming traditional kernel methods and classifier-based tests in high-dimensional, complex distributions.

ABSTRACT

We propose a class of kernel-based two-sample tests, which aim to determine whether two sets of samples are drawn from the same distribution. Our tests are constructed from kernels parameterized by deep neural nets, trained to maximize test power. These tests adapt to variations in distribution smoothness and shape over space, and are especially suited to high dimensions and complex data. By contrast, the simpler kernels used in prior kernel testing work are spatially homogeneous, and adaptive only in lengthscale. We explain how this scheme includes popular classifier-based two-sample tests as a special case, but improves on them in general. We provide the first proof of consistency for the proposed adaptation method, which applies both to kernels on deep features and to simpler radial basis kernels or multiple kernel learning. In experiments, we establish the superior performance of our deep kernels in hypothesis testing on benchmark and real-world data. The code of our deep-kernel-based two sample tests is available at https://github.com/fengliu90/DK-for-TST.

Motivation & Objective

• Motivate non-parametric two-sample testing in high-dimensional and complex-structure data.
• Propose a kernel that combines deep feature learning with a kernel on features and a stabilizing term on inputs.
• Maximize test power by optimizing a regularized MMD-based criterion.
• Provide theoretical guarantees for consistency of the kernel adaptation procedure.
• Empirically demonstrate superior performance on synthetic, physics, and image datasets.

Proposed method

• Define a deep kernel k_ω that combines a deep feature map φ_ω with a simple kernel on features and a stabilizing input kernel: k_ω(x,y) = [(1−ε) κ(φ_ω(x), φ_ω(y)) + ε] q(x,y).
• Optimize the kernel by maximizing the regularized power criterion Ŵ_J(ω) = MMD_u^2(S_P,S_Q;k_ω) / σ̂_H1,λ, where σ̂_H1,λ is a regularized variance estimate.
• Train φ_ω and kernel parameters ω via minibatch stochastic optimization to maximize Ŵ_J using Adam.
• Split data into training and testing to avoid data leakage; select the kernel on training data and evaluate power on held-out test data.
• Provide a permutation-based test to obtain p-values and thresholds for the learned kernel.
• Show that deep kernels are characteristic and provide consistency results for the kernel adaptation procedure.

Experimental results

Research questions

• RQ1 Can a kernel learned via a deep neural network provide higher test power for MMD-based two-sample tests than fixed kernels?
• RQ2 Does optimizing the power criterion with regularization yield a kernel that generalizes to unseen data?
• RQ3 How do deep kernels compare to classifier-based two-sample tests and other kernel-based methods on complex, high-dimensional distributions?
• RQ4 What are the theoretical guarantees for consistency and selection of the best kernel in this adaptation framework?

Key findings

• Learned deep kernels (MMD-D) achieve higher empirical test power than deep-kernel alternatives with fixed parameters and standard MMD in benchmark and real data.
• MMD-D outperforms simple Gaussian-lengthscale optimization (MMD-O) and classifier-based tests (C2ST-S, C2ST-L) across several datasets, especially at higher sample regimes.
• The proposed Ĵ_λ criterion guides kernel selection by balancing MMD with variance, and the authors establish consistency results for the kernel adaptation.
• The framework encompasses classifier-based two-sample tests as a special case, but deep kernels offer more general and efficient representations.
• Across datasets like Blob, Higgs, MNIST vs DCGAN samples, and high-dimensional Gaussian mixtures, MMD-D shows superior test power with controlled type I error.