[Paper Review] Refactor Analysis: Predictive Evaluations of Factor Models and Dimensionality

Michael Hardy|arXiv (Cornell University)|Mar 21, 2026

Psychometric Methodologies and Testing0 citations

TL;DR

The paper introduces Refactor and Verifactor analyses to evaluate unidimensionality as data recoverability rather than image fit, comparing multiple association measures and emphasizing out-of-sample generalization.

ABSTRACT

Unidimensional factor models justify some of the most consequential summaries in science -- single scores, single ranks, and single leaderboards -- yet unidimensionality is usually assessed indirectly by fitting and evaluating models on images of the data (e.g., correlation matrices) rather than on the response matrix itself. We introduce Refactor analysis, a data-first evaluation paradigm that converts a one-factor solution into a rank-1 prediction of the original matrix by estimating both respondent- and item-side structure from dual association images. We further introduce Verifactor analysis, which evaluates the same construction under bi-cross-validated (BCV) row-column partitions for improved generalization. In simulations where the data-generating mechanism is truly rank-1 and correlational, Refactor metrics align with classical unidimensionality indices, validating the approach. However, across 200 public dichotomous datasets, traditional fit and unidimensionality measures, though highly intercorrelated, are weakly related to data recoverability, especially out of sample. This gap exposes a methodological vulnerability: excellent image-based fit can coexist with poor data-level explanatory power. Finally, treating the association measure itself as a testable hypothesis, we compare $ϕ$, tetrachoric, and quadrant correlation, $q^\prime$, an important reintroduction. Quadrant correlation emerges as a simple, interpretable, and remarkably robust alternative, yielding consistently stronger reconstruction and more stable behavior under sample-size variation than commonly used correlations. Together, Refactor and Verifactor shift unidimensionality assessment from "does a one-factor model fit the correlation matrix?" to the question that matters for measurement and benchmarking: does a one-factor dependence structure recover and generalize the observed responses?

Motivation & Objective

• Motivate unidimensionality assessment as a recoverability problem rather than solely image fit.
• Introduce Refactor analysis to reconstruct the data matrix from rank-1 representations derived from association images.
• Introduce Verifactor analysis with bi-cross-validated (BCV) partitions to evaluate out-of-sample generalization on held-out rows and columns.
• Compare different association measures (e.g., Pearson, tetrachoric, quadrant) in the Refactor/Verifactor framework.
• Provide guidance on interpreting traditional image-based unidimensionality metrics alongside data-level recoverability.

Proposed method

• Define Refactor reconstruction as a rank-1 outer product using row- and column-side loadings derived from association images.
• Construct A_r and A_c as row- and column-wise association images from the data X.
• Obtain loadings u and v via a dimensionality-reduction operator on A_r and A_c respectively.
• Reconstruct X as X_hat = u v^T and evaluate m(X, X_hat) to assess recoverability.
• Extend to Verifactor by using bi-cross-validated blocks to hold out rows and columns and predict held-out blocks as tilde{A}_{ij}^{(A)}, averaging over folds.
• Employ isotonic R^2 as a data-based recovery metric and ECV as an image-based unidimensionality metric for comparison.
• Discuss the role of different association operators A (e.g., Pearson, tetrachoric, quadrant) in recoverability.

Experimental results

Research questions

• RQ1 Does a one-factor model recover and generalize the observed responses beyond image fit?
• RQ2 How do Refactor and Verifactor metrics align or diverge from traditional image-based unidimensionality indices like ECV?
• RQ3 Which association measures best capture the latent structure for reconstruction and prediction in binary/ordinal data?
• RQ4 Can isotonic R^2 serve as a monotone, interpretable recovery measure for rank-1 representations?
• RQ5 What is the impact of using BCV block prediction on assessing generalization across random rows and columns?

Key findings

• Refactor metrics align with classical unidimensionality indices in rank-1 simulated data settings but reveal divergences in real data between image fit and data recoverability.
• Across 200 public dichotomous datasets, traditional fit measures are weakly related to data recoverability, highlighting a vulnerability of image-based assessments.
• Quadrant correlation emerges as a robust alternative to Pearson and tetrachoric correlations for reconstructing and predicting data in the Refactor/Verifactor framework.
• Isotonic R^2 provides a monotone, variance-explained recovery metric that is optimal for monotone relationships between latent signals and observed responses.
• Verifactor with BCV targets true out-of-sample generalization in random-row/random-column designs, reducing leakage and optimistic bias in assessment.
• The framework shifts unidimensionality testing from image fit to recoverability and generalization of the rank-1 structure.