[Paper Review] Revisiting Relative Indicators and Provisional Truths

Loet Leydesdorff, Tobias Opthof|arXiv (Cornell University)|Aug 29, 2018

scientometrics and bibliometrics research19 references4 citations

TL;DR

This paper advocates for advancing citation impact indicators beyond current statistical methods by revisiting relative indicators and provisional truths in scientometric evaluation. It proposes integrating non-parametric approaches—such as top-10% citation thresholds—into impact assessment to better handle skewed citation distributions, offering a more robust alternative to traditional z-scores and expected citation ratios.

ABSTRACT

Following discussions in 2010 and 2011, scientometric evaluators have increasingly abandoned relative indicators in favor of comparing observed with expected citation ratios. The latter method provides parameters with error values allowing for the statistical testing of differences in citation scores. A further step would be to proceed to non-parametric statistics (e.g., the top-10%) given the extreme skewness (non-normality) of the citation distributions. In response to a plea for returning to relative indicators in the previous issue of this newsletter, we argue in favor of further progress in the development of citation impact indicators.

Motivation & Objective

• To re-evaluate the use of relative indicators in scientometric evaluation after their decline in favor of statistical z-scores.
• To address the limitations of parametric statistics in citation analysis due to extreme skewness in citation distributions.
• To propose a shift toward non-parametric methods such as top-10% citation thresholds for more reliable impact assessment.
• To respond to calls for a return to relative indicators by offering a methodologically advanced alternative grounded in statistical robustness.
• To improve the validity and interpretability of citation impact indicators in research evaluation contexts.

Proposed method

• Re-evaluating the use of relative indicators by comparing observed citation counts with expected citation ratios.
• Applying statistical testing to differences in citation scores using error values derived from expected citation models.
• Advocating for non-parametric statistics due to the non-normal distribution of citation counts.
• Introducing the top-10% citation threshold as a robust alternative to parametric indicators.
• Emphasizing the use of provisional truths in citation analysis to reflect uncertainty and context-dependency.
• Integrating methodological rigor with epistemological reflection on the nature of citation impact metrics.

Experimental results

Research questions

• RQ1 How can citation impact indicators be improved to better reflect the non-normal distribution of citations?
• RQ2 What are the limitations of using z-scores and expected citation ratios in scientometric evaluation?
• RQ3 To what extent can non-parametric methods like the top-10% threshold enhance the reliability of citation impact assessment?
• RQ4 Why has the shift from relative indicators to statistical ratios led to methodological shortcomings in research evaluation?
• RQ5 How can the concept of 'provisional truths' improve the interpretability of citation metrics in academic evaluation?

Key findings

• The use of relative indicators has been unduly marginalized in favor of parametric statistical methods that assume normality.
• Citation distributions are highly skewed, making parametric tests like z-scores statistically inappropriate without transformation.
• Non-parametric approaches such as the top-10% citation threshold offer a more robust and interpretable alternative to traditional indicators.
• Expected citation ratios with error values provide a statistical foundation, but do not resolve the issue of non-normality.
• The concept of 'provisional truths' supports a more nuanced, context-sensitive interpretation of citation impact metrics.
• A methodological shift toward non-parametric indicators is both necessary and feasible for improving research evaluation practices.