[Paper Review] The piranha problem: Large effects swimming in a small pond

Christopher Tosh, Philip Greengard+4 more|arXiv (Cornell University)|May 27, 2021

Philosophy and History of Science12 citations

TL;DR

This paper introduces the 'piranha problem'—a theoretical constraint showing that in multivariate systems, numerous large effects on a single outcome are statistically implausible unless explanatory variables are highly interdependent. Using correlation, regression, and mutual information theorems, it demonstrates that such effects would require implausible interactions or dependencies, challenging the validity of many social science findings with purported large, independent effects.

ABSTRACT

In some scientific fields, it is common to have certain variables of interest that are of particular importance and for which there are many studies indicating a relationship with different explanatory variables. In such cases, particularly those where no relationships are known among the explanatory variables, it is worth asking under what conditions it is possible for all such claimed effects to exist simultaneously. This paper addresses this question by reviewing some theorems from multivariate analysis showing that, unless the explanatory variables also have sizable dependencies with each other, it is impossible to have many such large effects. We discuss implications for the replication crisis in social science.

Motivation & Objective

• To investigate the theoretical limits on how many large effects can coexist on a single outcome variable in multivariate systems.
• To challenge the plausibility of numerous large, independent effects reported in social science research, especially in light of the replication crisis.
• To formalize mathematical constraints on correlation, regression coefficients, and mutual information that limit the number of large effects.
• To provide a theoretical framework for evaluating clusters of studies claiming large, independent effects on a single outcome.
• To highlight the role of variable dependencies and interactions in making large, stable effects statistically coherent.

Proposed method

• Derives mathematical theorems from multivariate analysis to constrain the number of large effects in a system with fixed outcome variance.
• Applies these theorems to correlation, linear regression, and mutual information to show that large effects require strong dependencies or interactions.
• Uses R² decomposition to illustrate that the total explained variance cannot exceed 1, limiting the number of large effects.
• Analyzes nonlinear models (e.g., logistic regression) to show that large independent effects lead to unrealistic probability predictions.
• Proposes that strong negative interactions or high negative correlations among predictors are required to sustain multiple large effects, which is unlikely in typical social science data.
• Draws analogies to the 'bet on sparsity' principle and hierarchical priors (e.g., horseshoe) as practical solutions to the piranha problem.

Experimental results

Research questions

• RQ1 Under what conditions can multiple large effects on a single outcome coexist in a multivariate system?
• RQ2 How do dependencies among explanatory variables affect the plausibility of large, independent effects in regression models?
• RQ3 What constraints do variance decomposition and R² place on the number of large effects in a system?
• RQ4 Why are large, independent effects in social science research often implausible due to interaction effects?
• RQ5 How do nonlinear models like logistic regression further constrain the existence of multiple large, independent effects?

Key findings

• It is mathematically implausible for a large number of explanatory variables to have large, independent effects on a single outcome unless they are highly interdependent.
• The total R² of a regression model cannot exceed 1, which limits the number of large effects unless they are correlated or interact strongly.
• In nonlinear models such as logistic regression, multiple large independent effects can drive predicted probabilities to extreme values (e.g., 0.01 to 0.99), which is unrealistic in applied settings.
• The only way to sustain multiple large effects without violating variance or probability constraints is through strong negative interactions or high negative correlations, which are uncommon in social science data.
• Theoretical constraints suggest that many widely reported large effects in social science may be overstated or non-replicable due to unaccounted dependencies and interactions.
• The piranha problem provides a theoretical basis for skepticism toward clusters of studies claiming large, independent effects, especially when explanatory variables show low inter-correlation.