[Paper Review] Ranking and mapping of universities and research-focused institutions worldwide based on highly-cited papers: A visualization of results from multi-level models

Lutz Bornmann, Moritz Stefaner+2 more|arXiv (Cornell University)|Dec 3, 2012

scientometrics and bibliometrics research40 references4 citations

TL;DR

This paper presents a web-based tool that ranks and maps global universities and research institutions using multi-level models to analyze highly-cited papers from Scopus data. By applying hierarchical modeling to estimate institution-specific citation impact with confidence intervals, it enables statistically robust comparisons of research excellence across fields and geographies, offering a more accurate alternative to raw citation counts for identifying centers of excellence.

ABSTRACT

The web application presented in this paper allows for an analysis to reveal centres of excellence in different fields worldwide using publication and citation data. Only specific aspects of institutional performance are taken into account and other aspects such as teaching performance or societal impact of research are not considered. Based on data gathered from Scopus, field-specific excellence can be identified in institutions where highly-cited papers have been frequently published. The web application combines both a list of institutions ordered by different indicator values and a map with circles visualizing indicator values for geocoded institutions. Compared to the mapping and ranking approaches introduced hitherto, our underlying statistics (multi-level models) are analytically oriented by allowing (1) the estimation of values for the number of excellent papers for an institution which are statistically more appropriate than the observed values; (2) the calculation of confidence intervals as measures of accuracy for the institutional citation impact; (3) the comparison of a single institution with an "average" institution in a subject area, and (4) the direct comparison of at least two institutions.

Motivation & Objective

• To develop a statistically rigorous method for assessing institutional research excellence based on highly-cited papers.
• To address limitations of raw citation counts by incorporating statistical modeling to reduce noise and improve reliability.
• To enable direct comparison of institutions within specific academic fields using standardized metrics.
• To visualize global research performance through interactive maps and ranked lists for field-specific excellence.
• To provide confidence intervals for institutional citation impact, enhancing transparency and interpretability of rankings.

Proposed method

• Application of multi-level (hierarchical) models to estimate the number of highly-cited papers per institution, adjusting for field and publication year effects.
• Use of field-normalized citation data from Scopus to identify papers in the top 10% of citations within their subject area and publication year.
• Estimation of institution-specific citation impact using empirical Bayes shrinkage, reducing extreme values from small institutions.
• Computation of 95% confidence intervals for each institution’s estimated citation impact to assess precision.
• Integration of geocoded data to create interactive maps with circles representing institutional performance levels.
• Implementation of a web application allowing users to compare institutions directly and view rankings by field and indicator.

Experimental results

Research questions

• RQ1 How can institutional research performance be measured more accurately than by raw citation counts?
• RQ2 To what extent do multi-level models improve the reliability of citation-based rankings for universities and research institutions?
• RQ3 Can confidence intervals for citation impact enhance the interpretability of institutional performance metrics?
• RQ4 How do institutions compare in terms of field-normalized citation impact across different global regions and disciplines?
• RQ5 What are the most effective ways to visualize and rank research excellence at scale using citation data?

Key findings

• Multi-level models significantly reduce the influence of random variation in citation counts, especially for smaller institutions with limited publication output.
• The estimated number of highly-cited papers per institution is more stable and reliable than observed raw counts, particularly when sample sizes are small.
• Confidence intervals provide a clear measure of uncertainty, allowing users to assess the reliability of an institution’s performance ranking.
• Institutions with high citation impact are consistently identified across multiple fields, revealing distinct global centers of excellence.
• The interactive web application enables direct comparison of at least two institutions, supporting evidence-based institutional benchmarking.
• Field-specific excellence is effectively visualized through geocoded maps, highlighting regional clusters of high-performing institutions.