[Paper Review] Identifying Causes of Test Unfairness: Manipulability and Separability

Youmi Suk, Weicong Lyu|arXiv (Cornell University)|Jan 19, 2026

Psychometric Methodologies and Testing0 citations

TL;DR

The paper introduces a causal framework for DIF based on treatment decomposition and separable effects to identify actionable, intervenable sources of test unfairness in non-manipulable group variables like gender and ELL status, and demonstrates detection via causal forests and BART.

ABSTRACT

Differential item functioning (DIF) is a widely used statistical notion for identifying items that may disadvantage specific groups of test-takers. These groups are often defined by non-manipulable characteristics, e.g., gender, race/ethnicity, or English-language learner (ELL) status. While DIF can be framed as a causal fairness problem by treating group membership as the treatment variable, this invokes the long-standing controversy over the interpretation of causal effects for non-manipulable treatments. To better identify and interpret causal sources of DIF, this study leverages an interventionist approach using treatment decomposition proposed by Robins and Richardson (2010). Under this framework, we can decompose a non-manipulable treatment into intervening variables. For example, ELL status can be decomposed into English vocabulary unfamiliarity and classroom learning barriers, each of which influences the outcome through different causal pathways. We formally define separable DIF effects associated with these decomposed components, depending on the absence or presence of item impact, and provide causal identification strategies for each effect. We then apply the framework to biased test items in the SAT and Regents exams. We also provide formal detection methods using causal machine learning methods, namely causal forests and Bayesian additive regression trees, and demonstrate their performance through a simulation study. Finally, we discuss the implications of adopting interventionist approaches in educational testing practices.

Motivation & Objective

• Motivate the need for a causal interpretation of DIF beyond non-manipulable group effects (e.g., gender, ELL).
• Propose an interventionist treatment-decomposition approach to identify separable sources of DIF.
• Define simple and general separable DIF and establish identification strategies under a FFRCISTG framework.
• Illustrate the framework with SAT and Regents exam items and discuss practical implications for testing practice.

Proposed method

• Adopt treatment decomposition to split non-manipulable treatments into intervening components that operate via distinct causal pathways.
• Define separable direct and indirect effects (SDE and SIE) and their conditional forms using SWIGs and potential outcomes.
• Extend to simple separable DIF (no item impact) and general separable DIF (with item impact) with corresponding identification formulas.
• Derive identification formulas under consistency, ignorability, positivity, and dismissible component conditions (future trial G).
• Propose detection methods using causal forests and Bayesian additive regression trees (BART).
• Apply framework to real items from SAT math and Regents math exams and conduct simulation studies to assess performance.

Experimental results

Research questions

• RQ1 What are the separable causal sources of DIF that arise from decomposing non-manipulable group treatments?
• RQ2 How can separable DIF be defined and identified when item impact is absent or present?
• RQ3 Can causal machine learning methods (causal forests, BART) effectively detect separable DIF in practice?
• RQ4 What do the SAT and Regents examples reveal about actionable sources of test unfairness under interventionist decomposition?

Key findings

• Introduces separable DIF as the difference in item functioning between two intervenable worlds defined by decomposed treatment components.
• Provides nonparametric identification strategies for simple and general separable DIF under FFRCISTG with explicit assumptions.
• Demonstrates detection methods using causal forests and BART and evaluates them through a simulation study.
• Applies the framework to biased SAT and Regents items to illustrate real-world interpretability of separable DIF components.
• Argues for interventionist approaches to identify and rectify actionable causes of test unfairness in educational testing.