[Paper Review] Probabilistic Models for Computerized Adaptive Testing: Experiments

Martin Plajner, Jiří Vomlel|arXiv (Cornell University)|Jan 28, 2016

Intelligent Tutoring Systems and Adaptive Learning3 references3 citations

TL;DR

This paper evaluates probabilistic models—Item Response Theory (IRT), Bayesian networks (BNs), and neural networks (NNs)—for computerized adaptive testing (CAT) using empirical data from 281 grammar school students. It employs a 10-fold cross-validation simulation to compare model performance, finding that the new Bayesian network model outperforms both IRT and neural network approaches in estimating student abilities with higher success rates in predicting responses.

ABSTRACT

This paper follows previous research we have already performed in the area of Bayesian networks models for CAT. We present models using Item Response Theory (IRT - standard CAT method), Bayesian networks, and neural networks. We conducted simulated CAT tests on empirical data. Results of these tests are presented for each model separately and compared.

Motivation & Objective

• To evaluate and compare the performance of three probabilistic models—IRT, Bayesian networks, and neural networks—in computerized adaptive testing (CAT).
• To improve upon prior Bayesian network models for CAT by introducing new expert-structured BNs with enhanced representational capacity.
• To assess model effectiveness through simulated CAT procedures on real student test data using standardized evaluation metrics.
• To investigate whether non-IRT models can match or exceed the performance of the established IRT standard in adaptive testing.

Proposed method

• Conducted 10-fold cross-validation: 90% of data used to train each model, 10% used for testing in each fold.
• Simulated CAT procedures where questions were selected greedily based on information gain, with model updates after each response.
• Used marginal probability distributions over skill variables to predict the most likely answer to each question using MAP estimation: $ x_i^* = \arg\max_{x_i} P(X_i = x_i | \boldsymbol{S}) $.
• Evaluated models using a success ratio metric: $ \mathrm{SR}_s^t = \frac{1}{|\mathcal{X}|} \sum_{X_i \in \mathcal{X}} I(x_i^* = x_i') $, measuring prediction accuracy over all questions.
• For neural networks, trained a feedforward network with one hidden layer to predict student scores directly, using backpropagation with varying hidden layer sizes (3, 5, 7 neurons).
• Selected questions based on maximizing the variance of predicted scores: $ X^* = \arg\max_i \mathrm{Var}(SC | X_{i,x}) $, to maximize information gain.

Experimental results

Research questions

• RQ1 How do Bayesian network models compare to the standard IRT model in terms of response prediction accuracy in CAT?
• RQ2 Can neural network models effectively support adaptive testing when trained on real student response data?
• RQ3 Does the proposed new Bayesian network model with 7+1 hidden variables outperform previous BN models and the IRT baseline?
• RQ4 What impact does question selection strategy have on the convergence and accuracy of model predictions in adaptive testing?
• RQ5 Can non-IRT probabilistic models achieve comparable or better performance than IRT in estimating student abilities from minimal responses?

Key findings

• The new Bayesian network model with 7+1 hidden variables achieved the highest success rate in predicting student responses across all tested models.
• The IRT model performed well but was outperformed by the new BN model, particularly in early stages of testing.
• Neural network models showed the slowest improvement in response prediction accuracy, with success rates increasing gradually over time.
• The success ratio $ \mathrm{SR}_0 $, representing baseline prediction before any questions, was lowest for the neural network model, indicating weaker initial estimation capability.
• The new BN model demonstrated a significant performance advantage, suggesting that structured expert knowledge in BNs enhances estimation accuracy in adaptive testing.
• The study indicates that future improvements in BN models could be achieved through local structure modeling (e.g., deterministic conditional probability tables), reducing parameter count while increasing expressiveness.