[Paper Review] Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models

Jia Liu, Zhiyu Xu+1 more|arXiv (Cornell University)|Mar 16, 2026

Psychometric Methodologies and Testing0 citations

TL;DR

The paper develops a scalable cognitive-diagnosis framework for evaluating LLMs by learning a Q-matrix with embedding-informed priors and jointly estimating latent attributes and item parameters using SAEM, enabling fine-grained, scalable diagnosis on large benchmarks.

ABSTRACT

Large language models (LLMs) have achieved remarkable performance on diverse benchmarks, yet existing evaluation practices largely rely on coarse summary metrics that obscure underlying reasoning abilities. In this work, we propose novel methodologies to adapt cognitive diagnosis models (CDMs) in psychometrics to LLM evaluation, enabling fine-grained diagnosis via multidimensional discrete capability profiles and interpretable characterizations of LLM strengths and weaknesses. First, to enable CDM-based evaluation at benchmark scale (more than 1000 items), we propose a scalable method that jointly estimates LLM mastery profiles and the item-attribute Q-matrix, addressing key challenges posed by high-dimensional latent attributes (K > 20), large item pools, and the prohibitive computational cost of existing marginal maximum likelihood-based estimation. Second, we incorporate item-level textual information to construct AI-embedding-informed priors for the Q-matrix, stabilizing high-dimensional estimation while reducing reliance on costly human specification. We develop an efficient stochastic-approximation algorithm to jointly estimate LLM mastery profiles and the Q-matrix that balances data fit with text-embedding-informed priors. Simulation studies demonstrate accurate parameter recovery. An application to the MATH Level 5 benchmark illustrates the practical utility of our method for LLM evaluation and uncovers useful insights into LLMs' fine-grained capabilities.

Motivation & Objective

• Motivate moving beyond aggregate accuracy to structured, multidimensional capability profiles for LLM evaluation.
• Develop scalable Q-matrix learning that leverages item–text embeddings to inform prior structure.
• Propose a joint estimation algorithm that scales to high-dimensional K and large item pools J under N observeers.
• Provide theoretical consistency guarantees in high-dimensional, benchmark-scale regimes.
• Demonstrate practical utility on large mathematics benchmarks to reveal fine-grained LLM capabilities.

Proposed method

• Adopt the DINA cognitive diagnostic model to link item responses to a multidimensional latent attribute profile.
• Construct an embedding-informed reference Q-matrix Q^(R) from item-question–solution embeddings using BERTopic with UMAP and hierarchical clustering.
• Introduce a prior P(Q | Q^(R)) to guide Q-matrix estimation within a MAP framework.
• Develop a scalable stochastic-approximation EM (SAEM) algorithm that iteratively updates latent attributes A, Q, and item parameters (c, g) under the prior.
• Prove consistency results in a triple-asymptotic regime where N, J, and K diverge, with rates depending on item separation and prior strength.
• Demonstrate parameter recovery and Q-matrix learning performance in simulations with high K (15, 30) and large J (1000, 2000), and apply to MATH Level 5 benchmark with 2,765 LLMs and 903 items.

Experimental results

Research questions

• RQ1 Can a Q-matrix be learned at scale from response data augmented with embedding-based textual priors?
• RQ2 Does embedding-informed prior guidance improve stability and interpretability of high-dimensional CDMs in LLM evaluation?
• RQ3 Can a joint SAEM estimator recover latent attribute profiles and Q-matrix entries consistently when N, J, and K grow jointly?
• RQ4 How do LLMs' fine-grained capabilities organize into semantically meaningful attribute clusters on a large benchmark like MATH Level 5?

Key findings

• Embedding-derived Q^(R) partitions items into semantically coherent capability groups reflected in learned Q-matrices.
• The embedding-informed MAP estimator yields scalable updates for A and Q without enumerating 2^K latent profiles.
• SAEM scales linearly with N, J, and K and achieves accurate parameter recovery under high-dimensional regimes.
• Simulation studies show accurate Q-matrix and latent profile recovery for K=15 or 30 and J up to 2000 across various N.
• Application to MATH Level 5 with 2,765 LLMs and 903 items yields a 28-group Q-matrix with interpretable attribute labels and patterns like prior agreement, procedural augmentation, and structural reclassification.
• Theoretical consistency results (Theorem 1) establish recovery guarantees under high-dimensional, jointly growing N, J, K with explicit finite-sample rates.