[Paper Review] On Medians of (Randomized) Pairwise Means

Pierre Laforgue, Stéphan Clémençon+1 more|arXiv (Cornell University)|Nov 1, 2022

Statistical Methods and Inference17 references4 citations

TL;DR

This paper extends the Median-of-Means (MoM) framework to pairwise learning problems by introducing the Median of (Randomized) Pairwise Means (MoRPM), which estimates risks using medians of U-statistics computed over randomly sampled or partitioned data blocks. The method preserves the robustness and generalization bounds of MoM while enabling application to ranking, clustering, and metric learning under minimal assumptions.

ABSTRACT

Tournament procedures, recently introduced in Lugosi & Mendelson (2016), offer an appealing alternative, from a theoretical perspective at least, to the principle of Empirical Risk Minimization in machine learning. Statistical learning by Median-of-Means (MoM) basically consists in segmenting the training data into blocks of equal size and comparing the statistical performance of every pair of candidate decision rules on each data block: that with highest performance on the majority of the blocks is declared as the winner. In the context of nonparametric regression, functions having won all their duels have been shown to outperform empirical risk minimizers w.r.t. the mean squared error under minimal assumptions, while exhibiting robustness properties. It is the purpose of this paper to extend this approach in order to address other learning problems, in particular for which the performance criterion takes the form of an expectation over pairs of observations rather than over one single observation, as may be the case in pairwise ranking, clustering or metric learning. Precisely, it is proved here that the bounds achieved by MoM are essentially conserved when the blocks are built by means of independent sampling without replacement schemes instead of a simple segmentation. These results are next extended to situations where the risk is related to a pairwise loss function and its empirical counterpart is of the form of a $U$-statistic. Beyond theoretical results guaranteeing the performance of the learning/estimation methods proposed, some numerical experiments provide empirical evidence of their relevance in practice.

Motivation & Objective

• To extend the Median-of-Means (MoM) approach from standard regression to learning problems involving pairwise loss functions, such as ranking, clustering, and metric learning.
• To investigate the statistical performance of MoM when data blocks are constructed via independent sampling without replacement (SRSWoR) rather than deterministic partitioning.
• To establish theoretical guarantees for the Median of (Randomized) Pairwise Means (MoRPM) estimator in terms of deviation bounds and generalization error.
• To demonstrate that the robustness and convergence properties of MoM are preserved under randomized block construction, even when block sizes are random or variable.
• To provide empirical validation of the method’s effectiveness through numerical experiments on pairwise learning tasks.

Proposed method

• Proposes a new estimator, the Median of (Randomized) Pairwise Means (MoRPM), which computes the median of U-statistics over data blocks formed via independent SRSWoR or deterministic partitioning.
• Uses U-statistics of degree two to estimate pairwise risks, where each U-statistic is computed over a block of size $ B $, and the final estimate is the median of these block-level U-statistics.
• Applies Hoeffding's inequality and concentration bounds to control the deviation of the MoRPM estimator from the true risk $ \theta(h) $, leveraging the median's robustness to outliers.
• Derives high-probability deviation bounds by combining conditional concentration inequalities with the properties of U-statistics, particularly bounding the variance of block-level U-statistics.
• Introduces a two-stage analysis: first bounding the probability that a single block's U-statistic deviates from the true risk, then bounding the median's deviation using Hoeffding's inequality on the fraction of deviant blocks.
• Establishes that optimal block size $ B $ and number of blocks $ K $ can be chosen to minimize the deviation bound, with $ K \propto \log(1/\delta) $ and $ B \propto n/K $, ensuring sub-Gaussian concentration.

Experimental results

Research questions

• RQ1 Can the Median-of-Means (MoM) principle be extended to learning problems where the risk is defined as an expectation over pairs of observations, such as in pairwise ranking or metric learning?
• RQ2 Does randomization in block construction—via SRSWoR instead of deterministic partitioning—adversely affect the statistical performance of MoM-based estimators?
• RQ3 What are the finite-sample deviation bounds for the Median of (Randomized) Pairwise Means (MoRPM) estimator under general moment conditions?
• RQ4 How do the variance components of the U-statistic (first- and second-order projections) influence the convergence rate of the MoRPM estimator?
• RQ5 Can the MoRPM method achieve generalization error bounds comparable to those of standard MoM in nonparametric regression, even when the risk is a U-statistic of degree two?

Key findings

• The Median of (Randomized) Pairwise Means (MoRPM) estimator achieves a deviation bound of order $ \sqrt{\frac{\log(1/\delta)}{n}} + \sqrt{\frac{\log^2(1/\delta)}{n^2}} $, matching the optimal rate of standard MoM under minimal assumptions.
• When blocks are built via SRSWoR, the MoRPM estimator maintains the same high-probability deviation bounds as the partition-based MoM, proving that randomization does not degrade performance.
• The optimal choice of block size $ B $ and number of blocks $ K $ leads to a deviation bound with leading term $ \sqrt{C_1 \frac{\log(1/\delta)}{n}} $, where $ C_1 = 108\sigma_1^2(h) $, and a second-order term $ \sqrt{C_2 \frac{\log^2(1/\delta)}{n(2n - 9\log(1/\delta))}} $ with $ C_2 = 486\sigma_2^2(h) $.
• The method ensures that the final estimator is robust to heavy-tailed or contaminated data, as the median of block-level U-statistics reduces sensitivity to outliers.
• Theoretical analysis confirms that the MoRPM estimator converges to the true risk $ \theta(h) $ at a sub-Gaussian rate, even when the underlying loss function is not bounded.
• Numerical experiments confirm the practical relevance of MoRPM, showing competitive performance in pairwise learning tasks such as metric learning and ranking, especially under data contamination.