[Paper Review] Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models

Joshua C. Chang, Shashaank Vattikuti+1 more|arXiv (Cornell University)|Dec 5, 2019

Gaussian Processes and Bayesian Inference28 references4 citations

TL;DR

This paper proposes a unified, single-step probabilistic autoencoding framework for multidimensional item-response theory (IRT) that integrates sparsity-promoting horseshoe priors for within-IRT factorization, eliminating the need for pre-modeling factor analysis. By framing IRT as a probabilistic autoencoder with a Bayesian neural network encoder, the method enables interpretable, disentangled latent trait estimation with improved stability and dimensionality selection via WAIC, while preserving posterior dependencies for accurate scoring.

ABSTRACT

Item response theory (IRT) is a non-linear generative probabilistic paradigm for using exams to identify, quantify, and compare latent traits of individuals, relative to their peers, within a population of interest. In pre-existing multidimensional IRT methods, one requires a factorization of the test items. For this task, linear exploratory factor analysis is used, making IRT a posthoc model. We propose skipping the initial factor analysis by using a sparsity-promoting horseshoe prior to perform factorization directly within the IRT model so that all training occurs in a single self-consistent step. Being a hierarchical Bayesian model, we adapt the WAIC to the problem of dimensionality selection. IRT models are analogous to probabilistic autoencoders. By binding the generative IRT model to a Bayesian neural network (forming a probabilistic autoencoder), one obtains a scoring algorithm consistent with the interpretable Bayesian model. In some IRT applications the black-box nature of a neural network scoring machine is desirable. In this manuscript, we demonstrate within-IRT factorization and comment on scoring approaches.

Motivation & Objective

• To eliminate the need for pre-modeling exploratory factor analysis in multidimensional IRT by integrating factorization directly into the IRT model.
• To improve model stability and interpretability by using a hierarchical Bayesian framework with horseshoe priors for sparse coding of item-item relationships.
• To enable consistent scoring of new test respondents by coupling the IRT generative model with a Bayesian neural network encoder, forming a probabilistic autoencoder.
• To address the black-box nature of neural network scoring in high-stakes testing by preserving posterior dependencies through structured variational inference or MCMC.
• To support dimensionality selection in IRT using WAIC, ensuring model selection is grounded in predictive accuracy and uncertainty propagation.

Proposed method

• Uses a hierarchical Bayesian IRT model based on the graded response model (GRM) to model ordinal item responses with person-specific traits and item-specific parameters.
• Applies a horseshoe prior to induce sparsity in item-to-trait loadings, enabling automatic, data-driven factorization within the IRT model without external factor analysis.
• Frames the IRT model as a probabilistic decoder and couples it with a Bayesian neural network encoder to form a probabilistic autoencoder architecture.
• Employs variational inference (ADVI) with structured approximations or MCMC to preserve posterior dependencies during scoring, avoiding univariate mean-field approximations.
• Uses WAIC for model comparison and dimensionality selection, allowing principled comparison of models with different numbers of latent traits.
• Performs Monte Carlo sampling from the joint posterior to ensure accurate prediction distributions, especially when posterior dependencies are non-trivial.

Experimental results

Research questions

• RQ1 Can within-IRT factorization using sparse coding replace traditional pre-modeling factor analysis in multidimensional IRT?
• RQ2 How does integrating factorization directly into the IRT model affect the stability and interpretability of latent trait estimation?
• RQ3 To what extent does the probabilistic autoencoder framework improve scoring accuracy and consistency compared to standard IRT or black-box neural networks?
• RQ4 Can WAIC effectively guide dimensionality selection in high-dimensional IRT models with complex posterior dependencies?
• RQ5 How do the pooling properties of hierarchical Bayesian models enhance robustness in IRT when compared to linear factor analysis?

Key findings

• The within-IRT factorization using horseshoe priors produced more stable item loadings than linear factor analysis, particularly in cases where items switched factors entirely between models.
• The three-dimensional IRT model achieved slightly better predictive accuracy than the two-dimensional model, but the difference in WAIC was within one standard error, indicating similar performance.
• Items related to moral and religious beliefs, which were grouped into the first factor by linear factor analysis, were correctly reclassified into the second factor by within-IRT factorization, reflecting better alignment with content-based domain structure.
• The second dimension in the within-IRT model showed partial leakage into the third dimension, while linear factorization caused complete reassignment of items, demonstrating greater robustness of the Bayesian approach.
• The use of MCMC or structured variational inference was necessary to preserve posterior dependencies, as mean-field approximations led to incorrect prediction distributions.
• The probabilistic autoencoder framework enables fast, interpretable scoring via neural network inference while maintaining consistency with the Bayesian IRT model’s generative process.