[Paper Review] Reducing overfitting in challenge-based competitions

Elias Chaibub Neto, Bruce Hoff+11 more|arXiv (Cornell University)|Jul 1, 2016

Sports Analytics and Performance17 references3 citations

TL;DR

This paper proposes a Bayesian bootstrap-based variant of the Ladder algorithm to reduce overfitting in challenge-based competitions, especially with small datasets. By releasing bootstrapped estimates of leaderboard scores instead of raw performance, the method prevents participants from overfitting to minor fluctuations, effectively mitigating vulnerabilities exposed by Freedman’s paradox-inspired attacks while supporting arbitrary performance metrics.

ABSTRACT

Over-fitting is a dreaded foe in challenge-based competitions. Because participants rely on public leaderboards to evaluate and refine their models, there is always the danger they might over-fit to the holdout data supporting the leaderboard. The recently published Ladder algorithm aims to address this problem by preventing the participants from exploiting willingly or inadvertently minor fluctuations in public leaderboard scores during model refinement. In this paper, we report a vulnerability of the Ladder that induces severe over-fitting of the leaderboard when the sample size is small. To circumvent this attack, we propose a variation of the Ladder that releases a bootstrapped estimate of the public leaderboard score instead of providing participants with a direct measure of performance. We also extend the scope of the Ladder to arbitrary performance metrics by relying on a more broadly applicable testing procedure based on the Bayesian bootstrap. Our method makes it possible to use a leaderboard, with the technical and social advantages that it provides, even in cases where data is scant.

Motivation & Objective

• To address the vulnerability of the Ladder algorithm to overfitting in small-sample regression challenges.
• To prevent participants from exploiting minor fluctuations in public leaderboard scores during model refinement.
• To extend the Ladder mechanism to arbitrary performance metrics beyond those expressible as sample moments.
• To maintain the benefits of public leaderboards—such as iterative model refinement and community engagement—without compromising leaderboard integrity.
• To provide a robust, scalable solution for organizing fair and reliable challenge-based competitions in data-scarce domains like biomedical research.

Proposed method

• Replace the direct release of public leaderboard scores with bootstrapped estimates of performance to obscure minor score fluctuations.
• Use the Bayesian bootstrap to generate posterior distributions of performance differences (Δs) between successive model submissions.
• Apply a Bayesian hypothesis test with posterior odds (PO) to decide whether to release the new score or retain the previous best.
• For metrics not expressible as sample moments, use non-parametric bootstrap resampling to approximate the Bayesian posterior distribution.
• Implement a step-forward attack simulation based on Freedman’s paradox to test the Ladder’s vulnerability and validate the improved method.
• Integrate the method into a sequential model refinement pipeline where only statistically significant improvements are revealed to participants.

Experimental results

Research questions

• RQ1 Can the Ladder algorithm be made robust against adversarial overfitting attacks in small-sample regression settings?
• RQ2 How can the Ladder mechanism be extended to support arbitrary performance metrics, including non-linear and non-moment-based ones?
• RQ3 Does replacing direct leaderboard scores with bootstrapped estimates effectively reduce overfitting without hindering model refinement?
• RQ4 To what extent does the Bayesian bootstrap-based approach preserve fairness and reliability in challenge-based competitions?
• RQ5 Can the method maintain the social and technical advantages of public leaderboards while preventing data leakage through score exploitation?

Key findings

• The original Ladder algorithm is vulnerable to overfitting in small-sample regression problems, particularly under Freedman’s paradox-inspired attacks.
• The proposed Bayesian bootstrap-based variant successfully prevents overfitting by obscuring minor score fluctuations through bootstrapped estimates.
• The method supports arbitrary performance metrics by leveraging the Bayesian bootstrap or non-parametric bootstrap as a proxy for posterior inference.
• The approach maintains model refinement capability while significantly reducing the risk of participants overfitting to public leaderboard data.
• Empirical validation using a step-forward attack demonstrates that the improved Ladder mechanism resists overfitting where the original Ladder fails.
• The method enables the use of public leaderboards in data-scarce domains such as biomedical research without compromising the integrity of final model evaluation.