[Paper Review] UniRank: A Multi-Agent Calibration Pipeline for Estimating University Rankings from Anonymized Bibliometric Signals

Pedram Riyazimehr, Seyyed Ehsan Mahmoudi|arXiv (Cornell University)|Feb 21, 2026

scientometrics and bibliometrics research0 citations

TL;DR

UniRank presents a three-stage multi-agent LLM pipeline that estimates global university rankings using anonymized bibliometric signals from OpenAlex and Semantic Scholar, with data-hiding to prevent memorization.

ABSTRACT

We present UniRank, a multi-agent LLM pipeline that estimates university positions across global ranking systems using only publicly available bibliometric data from OpenAlex and Semantic Scholar. The system employs a three-stage architecture: (a) zero-shot estimation from anonymized institutional metrics, (b) per-system tool-augmented calibration against real ranked universities, and (c) final synthesis. Critically, institutions are anonymized -- names, countries, DOIs, paper titles, and collaboration countries are all redacted -- and their actual ranks are hidden from the calibration tools during evaluation, preventing LLM memorization from confounding results. On the Times Higher Education (THE) World University Rankings ($n=352$), the system achieves MAE = 251.5 rank positions, Median AE = 131.5, PNMAE = 12.03%, Spearman $ρ= 0.769$, Kendall $τ= 0.591$, hit rate @50 = 20.7%, hit rate @100 = 39.8%, and a Memorization Index of exactly zero (no exact-match zero-width predictions among all 352 universities). The systematic positive-signed error (+190.1 positions, indicating the system consistently predicts worse ranks than actual) and monotonic performance degradation from elite tier (MAE = 60.5, hit@100 = 90.5%) to tail tier (MAE = 328.2, hit@100 = 20.8%) provide strong evidence that the pipeline performs genuine analytical reasoning rather than recalling memorized rankings. A live demo is available at https://unirank.scinito.ai .

Motivation & Objective

• Motivate estimating university rankings from publicly available bibliometric data without survey or proprietary signals.
• Propose a three-stage multi-agent architecture inspired by MAgICoRe for ranking estimation.
• Introduce an anonymization and data-hiding protocol to prevent memorization in LLMs.
• Develop a 16-feature bibliometric signal set from OpenAlex and Semantic Scholar.
• Provide a rigorous evaluation framework with novel Memorization Index and confidence intervals.

Proposed method

• Compute 16 bibliometric indicators from OpenAlex and Semantic Scholar for each institution.
• Anonymize identifying information while preserving numeric metrics.
• Stage 1: zero-shot estimation from anonymized metrics to produce rank ranges.
• Stage 2: per-system tool-augmented calibration using get_ranking_samples and compute_metrics to refine ranges.
• Stage 3: synthesize a final structured analysis report from Stage 1 and Stage 2 outputs.
• Use Wilson score intervals to report 95% confidence bounds and compute a Memorization Index (MI) to detect memorized predictions.

Figure 1 : UniRank system architecture. Data from OpenAlex and Semantic Scholar is aggregated, normalized, and anonymized before entering the three-stage LLM pipeline. During evaluation, the target university is hidden from the ranking store (dashed line) to prevent data leakage.

Experimental results

Research questions

• RQ1 Can a multi-agent LLM pipeline estimate university ranking positions from publicly available bibliometric data alone?
• RQ2 How effective is anonymization and data-hiding in preventing memorization while enabling reasoning-based estimation?
• RQ3 What is the contribution of calibration versus initial estimation to ranking accuracy across major systems (THE, QS, ARWU)?

Key findings

• On THE rankings (n=352), MAE is 251.5 positions and Spearman ρ is 0.769, with a Memorization Index of 0.000.
• Calibration yields a modest overall MAE improvement (256.8 to 251.5, -2.1%), with elite and tail tiers benefiting more.
• Hit@100 is 39.8% and hit@50 is 20.7% for THE, indicating meaningful ordinal alignment despite data limitations.
• Systematic positive signed error (+190.1 positions) shows the model underestimates rankings due to missing reputation and teaching signals.
• MI remains zero across tiers, supporting reasoning over memorization as the source of predictions.

Figure 2 : Three-stage pipeline: Stage 1 produces coarse zero-shot estimates from anonymized metrics. Stage 2 refines per-system with tool-augmented calibration (parallel). Stage 3 synthesizes the final report.