[Paper Review] Measuring science: irresistible temptations, easy shortcuts and dangerous consequences

Giovanni Abramo, Ciriaco Andrea D’Angelo|arXiv (Cornell University)|Dec 21, 2018

Evolution and Genetic Dynamics21 citations

TL;DR

This paper critiques the widespread misuse of total publication and citation counts in international research benchmarking, arguing that such metrics distort comparisons across disciplines due to inherent differences in scientific fertility. It demonstrates how failing to account for disciplinary heterogeneity leads to misleading rankings and calls for more nuanced, discipline-specific evaluation methods to avoid harmful policy and funding decisions.

ABSTRACT

In benchmarking international research, although publication and citation analyses should not be used to compare different disciplines, scientometrists frequently fail to resist the temptation to present rankings based on total publications and citations. Such measures are affected by significant distortions, due to the uneven fertility across scientific disciplines and the dishomogeneity of scientific specialisation among nations and universities. In this paper, we provide an indication of the extent of the distortions when comparative bibliometric analyses fail to recognise the range of levels of scientific fertility, not only within a given major disciplinary area but also within the different scientific disciplines encompassed by the same area.

Motivation & Objective

• To expose the dangers of using total publication and citation counts as proxies for research performance across diverse scientific disciplines.
• To highlight the temptation among scientometrists to use simple, easily accessible metrics despite their known limitations.
• To demonstrate how uneven scientific fertility across disciplines and institutions distorts comparative bibliometric analyses.
• To advocate for discipline-aware evaluation frameworks that account for inherent differences in research output and citation practices.
• To warn against the policy consequences of relying on misleading metrics in research funding and institutional ranking systems.

Proposed method

• Analyzing the structural disparities in publication and citation patterns across scientific disciplines and major research areas.
• Using empirical data from international research outputs to illustrate how total counts favor high-fertility fields like medicine and natural sciences over low-fertility fields like humanities.
• Comparing institutional and national rankings based on total publications and citations against discipline-normalized metrics.
• Highlighting the methodological flaws in cross-disciplinary benchmarking when fertility levels are not adjusted for.
• Drawing on real-world examples from research evaluation practices to illustrate the consequences of metric misuse.
• Proposing a shift toward discipline-specific normalization in bibliometric indicators to ensure fairer comparisons.

Experimental results

Research questions

• RQ1 Why do scientometric practices persist in using total publication and citation counts despite known disciplinary biases?
• RQ2 To what extent do differences in scientific fertility distort international research rankings when using aggregate metrics?
• RQ3 How do the uneven citation practices across disciplines affect the validity of cross-disciplinary performance comparisons?
• RQ4 What are the policy and institutional consequences of relying on simplistic bibliometric indicators?
• RQ5 What alternative evaluation frameworks could mitigate the distortions caused by aggregate metrics?

Key findings

• Total publication and citation counts are highly misleading for cross-disciplinary comparisons due to significant differences in scientific fertility across fields.
• Disciplines such as medicine and natural sciences have much higher publication and citation rates than fields like the humanities, leading to systematic bias in rankings.
• The use of aggregate metrics favors institutions and nations with strong representation in high-fertility fields, regardless of actual research quality or impact.
• Failure to normalize for disciplinary differences results in distorted institutional rankings and misallocation of research funding.
• The temptation to use easy, standardized metrics persists despite their known flaws, driven by administrative convenience and lack of methodological rigor.
• More nuanced, discipline-aware bibliometric methods are essential to ensure fair and accurate research evaluation.