[Paper Review] Fitting Bayesian item response models in Stata and Stan

Robert Grant, Bob Carpenter+2 more|arXiv (Cornell University)|Jan 13, 2016

Bayesian Modeling and Causal Inference3 references4 citations

TL;DR

This paper compares Bayesian item response modeling in Stata using bayesmh and StataStan, demonstrating that StataStan—powered by Stan’s Hamiltonian Monte Carlo and no-U-turn sampler—offers superior speed, efficiency, and scalability, especially for complex hierarchical Rasch models. StataStan achieves up to 24.1 times faster sampling per effective draw than bayesmh, particularly excelling in larger models and with parallel computing support.

ABSTRACT

Stata users have access to two easy-to-use implementations of Bayesian inference: Stata’s native bayesmh command and StataStan, which calls the general Bayesian engine, Stan. We compare these implementations on two important models for education research: the Rasch model and the hierarchical Rasch model. StataStan fits a more general range of models than can be fit by bayesmh and uses a superior sampling algorithm, that is, Hamiltonian Monte Carlo using the no-Uturn sampler. Furthermore, StataStan can run in parallel on multiple CPU cores, regardless of the flavor of Stata. Given these advantages and given that Stan is open source and can be run directly from Stata do-files, we recommend that Stata users interested in Bayesian methods consider using StataStan.

Motivation & Objective

• To evaluate and compare the performance of Stata’s native bayesmh and StataStan for fitting Bayesian item response models.
• To assess computational efficiency, sampling speed, and effective sample size across Rasch and hierarchical Rasch models.
• To determine whether StataStan’s advanced sampling algorithms and parallelization capabilities provide tangible advantages over bayesmh’s Metropolis-Hastings and Gibbs sampling.
• To provide practical guidance for Stata users on adopting StataStan for complex Bayesian modeling in education and social science research.

Proposed method

• Implementing the Rasch model and hierarchical Rasch model using both bayesmh and StataStan with identical data and priors.
• Using Hamiltonian Monte Carlo with the no-U-turn sampler (NUTS) in StataStan for efficient posterior sampling.
• Running four parallel chains in both software packages with identical initial values to ensure comparability.
• Measuring total time, simulation-only time, and effective sample size per second for key parameters (σ², τ², µ).
• Comparing results across model sizes (P = 100 to 10,000 persons) and evaluating the impact of random effects and parallelization.
• Using Stan’s default uniform priors for standard deviations and custom model specification via text-based Stan code in Stata.

Experimental results

Research questions

• RQ1 How do bayesmh and StataStan compare in fitting the Rasch model in terms of computational efficiency and effective sample size?
• RQ2 Does StataStan outperform bayesmh in sampling efficiency for hierarchical Rasch models, especially as model size increases?
• RQ3 To what extent does parallel computation in StataStan improve performance compared to serial execution and bayesmh’s limitations without Stata/MP?
• RQ4 How sensitive are the results to prior specification, and does StataStan’s default prior handling improve robustness?
• RQ5 What is the impact of using the random effects option in bayesmh on sampling efficiency and effective sample size?

Key findings

• StataStan was 2.4 to 14.1 times faster than bayesmh in total time and 3.5 to 24.1 times faster in simulation-only time for σ² in hierarchical Rasch models.
• For τ², StataStan was 0.6 to 2.8 times faster in total time and 0.8 to 6.9 times faster in simulation-only time.
• The µ hyperparameter was sampled more efficiently by bayesmh in most cases, though StataStan was up to 8.9 times faster in simulation-only time.
• Without the random effects option, bayesmh took 200 to 500 times longer to achieve the same effective sample size, particularly for σ² and τ².
• Parallel execution in StataStan reduced total time by nearly 50% (e.g., 734s to 383s for Rasch model with P=1000), while bayesmh could not use parallel chains without Stata/MP.
• StataStan’s model compilation overhead was outweighed by performance gains in larger models, making it more efficient overall.