[Paper Review] You Can't Publish Replication Studies (and How to Anyways)

Ghulam Jilani Quadri, Paul Rosen|arXiv (Cornell University)|Aug 23, 2019

Scientific Computing and Data Management38 references4 citations

TL;DR

This position paper addresses the challenge of publishing replication studies in visualization research, where novelty is prioritized over reproducibility. It proposes embedding replication work within novel studies through re-evaluation, expansion, or specialization—demonstrated via replications of Cleveland and McGill’s graphical perception study—showing how replication can be both scientifically rigorous and publishable by integrating it into original contributions with added value.

ABSTRACT

Reproducibility has been increasingly encouraged by communities of science in order to validate experimental conclusions, and replication studies represent a significant opportunity to vision scientists wishing contribute new perceptual models, methods, or insights to the visualization community. Unfortunately, the notion of replication of previous studies does not lend itself to how we communicate research findings. Simple put, studies that re-conduct and confirm earlier results do not hold any novelty, a key element to the modern research publication system. Nevertheless, savvy researchers have discovered ways to produce replication studies by embedding them into other sufficiently novel studies. In this position paper, we define three methods -- re-evaluation, expansion, and specialization -- for embedding a replication study into a novel published work. Within this context, we provide a non-exhaustive case study on replications of Cleveland and McGill's seminal work on graphical perception. As it turns out, numerous replication studies have been carried out based on that work, which have both confirmed prior findings and shined new light on our understanding of human perception. Finally, we discuss how publishing a true replication study should be avoided, while providing suggestions for how vision scientists and others can still use replication studies as a vehicle to producing visualization research publications.

Motivation & Objective

• To address the systemic difficulty of publishing replication studies in visualization and HCI due to the prioritization of novelty in peer review.
• To demonstrate that strict replication studies are rarely accepted, but can be successfully published by embedding them within novel research contributions.
• To provide actionable strategies—re-evaluation, expansion, and specialization—for researchers to integrate replication into original work while maintaining academic novelty.
• To advocate for a shift in research culture by showing how replication can coexist with innovation, especially at the intersection of vision science and visualization.
• To encourage reviewers and venues to recognize replication as a valuable component of scientific rigor, potentially rewarding it with 'bonus points' in peer review.

Proposed method

• Re-evaluation: Reassessing prior studies under different demographics, environments, or analytical methods to test robustness and reduce bias.
• Expansion: Extending original findings with new experiments that deepen understanding or reveal additional insights beyond the original scope.
• Specialization: Applying and refining prior findings to specific domains or contexts, leveraging domain-specific knowledge to generate novel contributions.
• Embedding replication within novel studies: Using replication as a foundation to build original methodological or conceptual innovations, ensuring the paper meets publication novelty criteria.
• Leveraging existing data and experimental designs from prior studies to generate new research questions without requiring new data collection.
• Using case studies of Cleveland and McGill’s graphical perception work to illustrate how replication can be systematically applied and validated across multiple replications.

Experimental results

Research questions

• RQ1 Why are replication studies rarely published in visualization and HCI despite their scientific value?
• RQ2 How can researchers successfully publish replication studies in a system that prioritizes novelty?
• RQ3 What strategies can embed replication into novel research contributions without compromising originality?
• RQ4 How do re-evaluation, expansion, and specialization enable replication to contribute meaningfully to new publications?
• RQ5 What role can replication play in strengthening foundational assumptions in visualization research?

Key findings

• Replication studies are systematically under-published in visualization and HCI due to the dominance of novelty in peer review, with only 3% of HCI papers attempting replication.
• Strict replication studies are rarely accepted for publication, as they are perceived as lacking novelty, similar to application papers in visualization.
• Re-evaluation, expansion, and specialization are effective strategies to embed replication into novel research, enabling publication while validating foundational findings.
• Numerous replications of Cleveland and McGill’s graphical perception study have been published, confirming original findings and revealing new insights into human perception and graphical encoding effectiveness.
• The integration of replication into novel works—such as applying findings to new domains or re-analyzing data with new methods—can produce high-impact, publishable research.
• Reviewers and venues could improve reproducibility by offering 'bonus points' for replication components, similar to how data availability improves review scores.