[Paper Review] Guaranteeing Reproducibility in Deep Learning Competitions

Brandon Houghton, Stephanie Milani+6 more|arXiv (Cornell University)|May 12, 2020

Adversarial Robustness in Machine Learning5 references4 citations

TL;DR

This paper proposes a novel competition paradigm for deep learning that guarantees reproducibility by retraining all final submissions under controlled conditions using a held-out test environment. By evaluating learning procedures rather than pre-trained agents and enforcing strict constraints on compute and data, the approach ensures robustness, generalization, and reduced exploitation of domain-specific knowledge, as demonstrated in the MineRL competition on sample-efficient reinforcement learning.

ABSTRACT

To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents. Since competition organizers re-train proposed methods in a controlled setting they can guarantee reproducibility, and -- by retraining submissions using a held-out test set -- help ensure generalization past the environments on which they were trained.

Motivation & Objective

• To address the lack of reproducibility and generalization in deep learning competitions due to overfitting to specific environments and undocumented modifications.
• To encourage the development of sample-efficient learning algorithms by shifting evaluation from pre-trained agents to learning procedures.
• To reduce reliance on domain-specific knowledge by requiring retraining in a perturbed environment with a new texture pack.
• To provide a scalable computational model for organizers by limiting retraining to top-performing teams in a second round.
• To improve participant engagement and confidence by offering both text and video demonstrations of successful submissions.

Proposed method

• Competition organizers retrain all final submissions in a controlled environment using a held-out test set with a new texture pack to ensure reproducibility and robustness.
• Participants are evaluated based on the performance of their learning procedures rather than pre-trained agents, promoting generalization beyond training environments.
• A large dataset of human demonstrations—both expert and non-expert—is provided to improve sample efficiency, with subtasks defined by simple metrics like time to completion.
• The dataset is re-simulated with visual changes to create a second, visually distinct but semantically equivalent dataset, enabling distribution shift testing.
• A two-round competition structure is used: first round allows broad participation, second round limits retraining to top ten teams to manage computational costs.
• Organizers enforce constraints on compute time and environment sample limits to ensure sample efficiency and prevent overfitting.

Experimental results

Research questions

• RQ1 How can deep learning competitions be structured to ensure reproducibility of results despite variations in training setup and environment configuration?
• RQ2 To what extent can learning procedures be evaluated independently of pre-trained models to promote generalization and robustness?
• RQ3 What role do demonstrations—especially non-expert or crowd-sourced ones—play in improving sample efficiency without requiring costly expert labeling?
• RQ4 How can competition organizers balance computational cost with the need to retrain submissions in a controlled, perturbed environment?
• RQ5 What impact do visual perturbations (e.g., new texture packs) have on the robustness of learned policies, and how can they be used to verify generalization?

Key findings

• Re-training final submissions in a new, previously unseen texture pack of the environment guarantees reproducibility and verifies robustness to visual distribution shifts.
• The two-round competition structure successfully limits computational overhead while still enabling rigorous re-evaluation of top-performing models.
• Providing both text and video demonstrations of successful submissions increased participant confidence and sustained engagement throughout the competition.
• Crowd-sourced demonstrations with simple subtask metrics enabled effective imitation learning without requiring expensive expert labeling.
• The use of re-simulated datasets with altered visual features allowed creation of semantically equivalent but visually distinct training and test sets, enhancing generalization testing.
• The paradigm successfully reduced exploitation of environment-specific knowledge, as policies that relied on such cues failed when tested on the new texture pack.