[Paper Review] Top Tips to Make Your Research Irreproducible

Neil Chue Hong, Tom Crick+3 more|arXiv (Cornell University)|Mar 31, 2015

Privacy-Preserving Technologies in Data1 references3 citations

TL;DR

This satirical paper humorously outlines tactics to intentionally make computational research irreproducible, such as omitting methodological details, using abstract pseudo-code, avoiding data sharing, and ensuring personal inability to reproduce results. The key contribution is a tongue-in-cheek manifesto exposing flaws in scientific publishing by parodying poor research practices.

ABSTRACT

It is an unfortunate convention of science that research should pretend to be reproducible; our top tips will help you mitigate this fussy conventionality, enabling you to enthusiastically showcase your irreproducible work.

Motivation & Objective

• To highlight the prevalence of irreproducibility in computational research by parodying common poor practices.
• To critique the scientific publishing culture that rewards publication over verifiability or transparency.
• To expose how vague methodologies, lack of code/data sharing, and over-reliance on citations perpetuate irreproducibility.
• To challenge researchers and reviewers to reflect on what constitutes rigorous, reproducible science by inverting the principles of reproducibility.

Proposed method

• Advocate omitting detailed experimental setup, suggesting only high-level, vague descriptions to obscure critical implementation details.
• Promote the use of abstract pseudo-code to communicate ideas without revealing underlying toolchains or manual interventions.
• Encourage omission of methodological limitations to save space, redirecting it toward citing influential papers to inflate h-index.
• Recommend creating proprietary, non-public benchmarks to avoid scrutiny, especially when methods underperform on standard datasets.
• Advise against sharing code or data to prevent others from verifying results or detecting flaws in implementation.
• Propose the core strategy of ensuring personal irreproducibility—deliberate carelessness to prevent even the author from reproducing results—thereby eliminating the risk of replication by others.

Experimental results

Research questions

• RQ1 How can researchers intentionally design studies to be irreproducible while still gaining acceptance in peer-reviewed journals?
• RQ2 What methodological omissions or obfuscations most effectively prevent replication without raising suspicion?
• RQ3 How does the absence of shared data, code, or detailed methods contribute to the persistence of false or unverified results?
• RQ4 In what ways does the current academic incentive structure reward irreproducibility over transparency?
• RQ5 What systemic flaws in peer review allow irreproducible research to be published and cited as if it were valid?

Key findings

• The paper demonstrates that irreproducibility is not only common but often facilitated by deliberate omissions in methodology, code, and data sharing.
• Using abstract pseudo-code and omitting implementation details effectively prevents others from understanding or replicating the actual computational process.
• Creating private, non-public benchmarks allows researchers to avoid validation on community-standard datasets, increasing the likelihood of unverifiable claims.
• The authors argue that if a researcher cannot reproduce their own work, the risk of external replication is eliminated, making the research effectively irreproducible by design.
• The paper concludes with a satirical but pointed claim: an irreproducible experiment is functionally equivalent to one that was never conducted at all, highlighting the absurdity of treating unverifiable results as valid science.