[Paper Review] Bayesian Item Response Modeling in R with brms and Stan

Paul‐Christian Bürkner|arXiv (Cornell University)|May 23, 2019

Mental Health Research Topics61 references103 citations

TL;DR

Demonstrates Bayesian IRT modeling in R using brms and Stan, enabling flexible, multilevel, distributional IRT specifications with covariates, priors, and model comparison.

ABSTRACT

Item Response Theory (IRT) is widely applied in the human sciences to model persons' responses on a set of items measuring one or more latent constructs. While several R packages have been developed that implement IRT models, they tend to be restricted to respective prespecified classes of models. Further, most implementations are frequentist while the availability of Bayesian methods remains comparably limited. We demonstrate how to use the R package brms together with the probabilistic programming language Stan to specify and fit a wide range of Bayesian IRT models using flexible and intuitive multilevel formula syntax. Further, item and person parameters can be related in both a linear or non-linear manner. Various distributions for categorical, ordinal, and continuous responses are supported. Users may even define their own custom response distribution for use in the presented framework. Common IRT model classes that can be specified natively in the presented framework include 1PL and 2PL logistic models optionally also containing guessing parameters, graded response and partial credit ordinal models, as well as drift diffusion models of response times coupled with binary decisions. Posterior distributions of item and person parameters can be conveniently extracted and post-processed. Model fit can be evaluated and compared using Bayes factors and efficient cross-validation procedures.

Motivation & Objective

• Introduce a flexible Bayesian IRT framework based on brms and Stan.
• Show how to specify and fit a wide range of IRT models within a multilevel formula syntax.
• Explain how to incorporate covariates, DIF, and distributional parameters into IRT models.
• Discuss prior choices for person/item parameters and model identification in a Bayesian context.
• Provide practical hands-on examples and guidance for applying the framework in real data.

Proposed method

• Use brms’ nonlinear/multilevel formula syntax to specify distributional IRT models.
• Leverage Stan for MCMC-based Bayesian estimation with adaptive Hamiltonian Monte Carlo.
• Support over 40 built-in response distributions and allow user-defined distributions.
• Model both fixed and random effects for persons and items, including covariate effects and interactions.
• Incorporate hierarchical (partial pooling) priors and LKJ priors for correlation structures.
• Provide approaches for model comparison (e.g., Bayes factors, cross-validation) and post-processing of posteriors.

Experimental results

Research questions

• RQ1 How can Bayesian IRT models be specified within a single, flexible framework using brms and Stan?
• RQ2 What range of response distributions and nonlinear/distributional IRT models can be implemented and estimated with this approach?
• RQ3 How can covariates, differential item functioning, and priors be incorporated to improve estimation and inference?
• RQ4 What are practical guidelines for model identification, prior choice, and model comparison in Bayesian IRT?
• RQ5 How can this framework be used in real data with hands-on examples to solve applied IRT questions?

Key findings

• The brms/Stan framework supports a wide variety of IRT models, including 1PL, 2PL, 3PL, GRM, PCM, RPCM, and drift-diffusion variants, across multiple response types.
• Users can specify nonlinear and distributional predictors, enabling flexible modeling where item discriminations and other parameters vary or depend on covariates.
• Priors can be set at person and item levels, with options for non-hierarchical and hierarchical (partial pooling) structures, including LKJ priors for correlations.
• The framework allows Bayesian model comparison using Bayes factors and efficient cross-validation for model assessment.
• Common IRT problems such as DIF can be explored via interaction terms and covariate selection methods within the Bayesian framework.
• All materials and examples are available on GitHub, and specific brms version requirements (2.11.5+) are provided for reproducing results.