[Paper Review] If your P value looks too good to be true, it probably is: Communicating reproducibility and variability in cell biology

Samuel J. Lord, Katrina Velle+2 more|arXiv (Cornell University)|Nov 8, 2019

Gene Regulatory Network Analysis28 references89 citations

TL;DR

The paper warns that using cell counts as independent samples yields misleading P values and offers practical tutorials to communicate cell-level variability and experimental reproducibility, including alternatives to P values.

ABSTRACT

The cell biology literature is littered with erroneously tiny P values, often the result of evaluating individual cells as independent samples. Because readers use P values and error bars to infer whether a reported difference would likely recur if the experiment were repeated, the sample size N used for statistical tests should actually be the number of times an experiment is performed, not the number of cells (or subcellular structures) analyzed across all experiments. P values calculated using the number of cells do not reflect the reproducibility of the result and are thus highly misleading. To help authors avoid this mistake, we provide examples and practical tutorials for creating figures that communicate both the cell-level variability and the experimental reproducibility.

Motivation & Objective

• Highlight the misuse of P values when cells are treated as independent samples.
• Demonstrate how cell-level variability differs from experimental reproducibility.
• Provide practical tutorials for creating figures that communicate variability and reproducibility.
• Advise on estimation statistics and alternative plots to P values in cell biology.

Proposed method

• Analyze how treating individual cells as independent samples inflates sample size and misleads P values.
• Provide examples illustrating the difference between cell-level variability and experimental reproducibility.
• Offer step-by-step tutorials for making plots in R, Python, and Excel to communicate variability and reproducibility.
• Discuss estimation statistics and forest plots as alternatives to relying solely on P values.

Experimental results

Research questions

• RQ1 How does counting cells as independent samples affect P value interpretation in cell biology experiments?
• RQ2 How can researchers effectively communicate cell-level variability and experimental reproducibility?
• RQ3 What plotting and statistical approaches (e.g., forest plots, estimation statistics) better convey reproducibility than traditional P values?

Key findings

• P values based on the number of cells, not experimental repetitions, can be highly misleading regarding reproducibility.
• The paper provides examples and practical tutorials to create figures that show both cell-level variability and experimental reproducibility.
• Estimation statistics and forest plots are proposed as reasonable alternatives or complements to P values.
• The authors clarify the benefits of P values while acknowledging their flaws and offer guidance to improve communication in figures.