[Paper Review] Equivalence testing for linear regression

Harlan Campbell|arXiv (Cornell University)|Apr 3, 2020

Advanced Statistical Methods and Models5 citations

TL;DR

This paper introduces equivalence testing procedures for linear regression, enabling researchers to statistically confirm the absence of meaningful associations between predictors and outcomes. It proposes tests for unstandardized regression coefficients and semipartial correlations using equivalence margins, with p-values and confidence intervals to assess negligible effects, offering a frequentist alternative to NHST and Bayesian methods in applied research.

ABSTRACT

We introduce equivalence testing procedures for linear regression analyses. Such tests can be very useful for confirming the lack of a meaningful association between a continuous outcome and a continuous or binary predictor. Specifically, we propose an equivalence test for unstandardized regression coefficients and an equivalence test for semipartial correlation coefficients. We review how to define valid hypotheses, how to calculate p-values, and how these tests compare to an alternative Bayesian approach with applications to examples in the literature.

Motivation & Objective

• To address the lack of established equivalence testing methods for linear regression, a widely used statistical technique.
• To enable researchers to formally test for the absence of meaningful associations between predictors and outcomes, overcoming the logical flaw of NHST in failing to confirm the null.
• To provide a frequentist alternative to null hypothesis significance testing (NHST) that allows researchers to conclude 'lack of meaningful effect' when appropriate.
• To offer practical tools for defining and justifying equivalence margins, especially when working with arbitrary or non-interpretive scales.
• To compare the proposed frequentist equivalence tests with Bayesian approaches using real-world examples from psychology and education research.

Proposed method

• Proposes an equivalence test for unstandardized regression coefficients using a two-one-sided t-test (TOST) approach with a defined equivalence margin Δ.
• Defines the null hypothesis as H₀: θ ∉ (−Δ, Δ), where θ is the regression coefficient, and tests whether the effect size is small enough to be considered negligible.
• Introduces equivalence testing for semipartial correlation coefficients (SPC) to assess the unique contribution of a predictor after controlling for others.
• Calculates p-values for equivalence tests using the TOST procedure, with confidence intervals for effect sizes to assess whether they fall entirely within the equivalence margin.
• Provides R functions (e.g., `equivSR`, `equiv_corrZ`) to compute p-values, effect sizes, and equivalence tests for both unstandardized and semipartial correlations.
• Compares results from frequentist equivalence tests with Bayesian factor approaches (e.g., Bayes factors) to evaluate consistency and interpretability across methods.

Experimental results

Research questions

• RQ1 Can we formally test for the absence of a meaningful association between a predictor and outcome in linear regression using equivalence testing?
• RQ2 How can researchers define and justify an equivalence margin Δ when working with arbitrary or non-interpretive scales?
• RQ3 How do the results of frequentist equivalence tests compare with Bayesian approaches in terms of p-values and Bayes factors for the same regression models?
• RQ4 What is the performance of equivalence testing in detecting negligible effects in real-world psychological and educational datasets?
• RQ5 How does controlling for covariates affect the outcome of equivalence tests in multiple regression models?

Key findings

• In the Salaries dataset, the equivalence test for the sex coefficient (β = 0.045) yielded a p-value of 0.076, failing to reject the null of meaningful effect at α = 0.05, suggesting the effect may be negligible.
• For the mindset study, the semipartial correlation between mindset score and learning goals was r = 0.098, with a p-value of 0.016 for equivalence testing, indicating the effect is likely negligible.
• In the mindset study, the correlation between mindset and performance-avoidance goals was r = -0.109, with a p-value of 0.027, suggesting the effect is within the equivalence margin of -0.20.
• For the Raven test score, the semipartial correlation with mindset score was r = -0.122, with a p-value of 0.000, indicating strong evidence for equivalence (i.e., negligible effect) within Δ = 0.20.
• When controlling for cognitive ability, the semipartial correlation between mindset and resilience (Raven test) dropped to r = -0.055, with a p-value of 0.000, confirming equivalence even after adjustment.
• Bayes factors in the mindset study showed strong evidence for the null in several cases (e.g., BF = 5.59e-21 for rank), aligning with frequentist equivalence test results.