[Paper Review] Unbiased estimators for the variance of MMD estimators

Danica J. Sutherland, Deka, Namrata|arXiv (Cornell University)|Jun 5, 2019

Statistical Methods and Inference5 references4 citations

TL;DR

This paper presents unbiased estimators for the variance of Maximum Mean Discrepancy (MMD) estimators, leveraging U-statistic theory to correct prior biased approaches. It derives computationally efficient, truly unbiased variance estimators for both the squared MMD and the difference of two correlated squared MMDs, with no additional cost beyond the original MMD computation.

ABSTRACT

The maximum mean discrepancy (MMD) is a kernel-based distance between probability distributions useful in many applications (Gretton et al. 2012), bearing a simple estimator with pleasing computational and statistical properties. Being able to efficiently estimate the variance of this estimator is very helpful to various problems in two-sample testing. Towards this end, Bounliphone et al. (2016) used the theory of U-statistics to derive estimators for the variance of an MMD estimator, and differences between two such estimators. Their estimator, however, drops lower-order terms, and is unnecessarily biased. We show in this note - extending and correcting work of Sutherland et al. (2017) - that we can find a truly unbiased estimator for the actual variance of both the squared MMD estimator and the difference of two correlated squared MMD estimators, at essentially no additional computational cost.

Motivation & Objective

• To address the lack of truly unbiased variance estimators for MMD estimators, which are critical for reliable two-sample testing.
• To correct the bias present in prior estimators that drop lower-order terms, especially problematic for small sample sizes.
• To extend and correct earlier work by [7] using U-statistic theory to derive exact variance expressions in terms of kernel matrices.
• To provide variance estimators that are unbiased and computationally efficient, matching the O(m²) complexity of the original MMD estimator.
• To clarify the trade-off between unbiasedness and practical performance, acknowledging that biased estimators may sometimes outperform in practice.

Proposed method

• Derives the variance of the unbiased MMD estimator using U-statistic theory, expressing it in terms of expectations of inner products in the RKHS.
• Introduces the uncentered covariance operator to express variance components in a compact, operator-based form.
• Constructs unbiased estimators for the variance components by replacing population expectations with empirical counterparts using kernel matrices.
• Derives explicit closed-form expressions for the variance estimator of the difference between two MMDs, applicable when comparing three distributions.
• Employs symmetric kernel matrices and vectorized operations (e.g., 1-norms, Frobenius norms, and matrix-vector products) to ensure O(m²) computational complexity.
• Validates the unbiasedness of the final estimators through theoretical derivation, ensuring they are consistent and free from systematic error.

Experimental results

Research questions

• RQ1 Can we derive a truly unbiased estimator for the variance of the squared MMD estimator, without dropping lower-order terms?
• RQ2 How can we extend unbiased variance estimation to the difference of two correlated squared MMD estimators, useful in three-sample testing?
• RQ3 What is the computational cost of achieving unbiased variance estimation compared to existing biased estimators?
• RQ4 Does the unbiased variance estimator outperform biased alternatives in practice, or are there trade-offs in finite-sample performance?
• RQ5 Can the theoretical framework be generalized to cases where sample sizes m and n are unequal?

Key findings

• The paper derives a fully unbiased estimator for the variance of the squared MMD estimator, correcting earlier biased approximations that dropped lower-order terms.
• The proposed estimator for the difference of two MMDs is unbiased and computable in O(m²) time, matching the complexity of the original MMD estimator.
• The final variance estimator (Equation 6) is expressed entirely in terms of kernel matrices and their operations, such as Frobenius norms, 1-norms of matrix products, and trace-like terms.
• Despite being unbiased, the paper acknowledges that in practice, simpler biased estimators (e.g., from [4]) may perform better due to lower variance, as confirmed by experiments in [2].
• The method generalizes to unequal sample sizes m and n, improving on prior work that assumed equal or symmetric samples.
• The derivation confirms that the unbiased estimator avoids the bias introduced by omitting lower-order terms, which can significantly affect accuracy in small samples.