[Paper Review] The disruption index is biased by citation inflation

Alexander M. Petersen, Felber Arroyave+1 more|arXiv (Cornell University)|Jun 2, 2023

scientometrics and bibliometrics researchDecision Sciences3 citations

TL;DR

This paper demonstrates that the disruption index (CD) is systematically biased by citation inflation—driven by longer reference lists and rising self-citations—causing it to erroneously decline over time. The authors show this bias invalidates cross-temporal comparisons and propose policy interventions like reference list caps to stabilize evaluation metrics.

ABSTRACT

A recent analysis of scientific publication and patent citation networks by Park et al. (Nature, 2023) suggests that publications and patents are becoming less disruptive over time. Here we show that the reported decrease in disruptiveness is an artifact of systematic shifts in the structure of citation networks unrelated to innovation system capacity. Instead, the decline is attributable to 'citation inflation', an unavoidable characteristic of real citation networks that manifests as a systematic time-dependent bias and renders cross-temporal analysis challenging. One driver of citation inflation is the ever-increasing lengths of reference lists over time, which in turn increases the density of links in citation networks, and causes the disruption index to converge to 0. A second driver is attributable to shifts in the construction of reference lists, which is increasingly impacted by self-citations that increase in the rate of triadic closure in citation networks, and thus confounds efforts to measure disruption, which is itself a measure of triadic closure. Combined, these two systematic shifts render the disruption index temporally biased, and unsuitable for cross-temporal analysis. The impact of this systematic bias further stymies efforts to correlate disruption to other measures that are also time-dependent, such as team size and citation counts. In order to demonstrate this fundamental measurement problem, we present three complementary lines of critique (deductive, empirical and computational modeling), and also make available an ensemble of synthetic citation networks that can be used to test alternative citation-based indices for systematic bias.

Motivation & Objective

• To identify and diagnose systematic bias in the disruption index (CD) due to citation inflation in scientific citation networks.
• To demonstrate that the observed decline in CD over time is an artifact of structural changes in citation networks, not actual reductions in scientific disruption.
• To challenge the validity of using CD for cross-temporal analysis in research evaluation, especially when correlated with time-dependent variables like team size or citation counts.
• To propose policy interventions—such as reference list caps—to mitigate citation inflation and stabilize bibliometric metrics.
• To provide synthetic citation networks for testing alternative, bias-resistant citation-based indices.

Proposed method

• Conduct a deductive critique of the disruption index's mathematical structure, identifying how increasing reference list length (r_p) and triadic closure from self-citations introduce time-dependent bias.
• Perform empirical analysis on real citation networks (e.g., Microsoft Academic Graph) to show rising r_p and citation counts over time, correlating with declining CD values.
• Use computational modeling to simulate citation network growth under varying parameters (e.g., r_p growth, self-citation rates), demonstrating how CD converges to zero due to citation inflation.
• Apply regression models with fixed effects for publication year to assess relationships between CD and variables like team size (k_p), revealing spurious correlations due to time confounding.
• Develop and release an ensemble of synthetic citation networks (DryadDisruption2023) to enable testing of alternative bibliometric indices for systematic bias.
• Introduce a policy simulation framework to evaluate the impact of reference list caps on reducing citation inflation effects.

Figure 1: ‘Citation inflation’ attributable to the increasing number and length of reference lists. (a) Schematic illustrating the inflation of the reference supply owing to the fact that the annual publication rate $n(t)$ (comprised of increasing diversity of article lengths), along with the number

Experimental results

Research questions

• RQ1 To what extent does the observed decline in the disruption index (CD) over time reflect actual changes in scientific innovation, or is it an artifact of citation inflation?
• RQ2 How do increasing reference list lengths and rising self-citation rates contribute to the temporal bias in the disruption index?
• RQ3 Can the disruption index be reliably used for cross-temporal comparisons in research evaluation, given the time-dependent structural shifts in citation networks?
• RQ4 What policy interventions, such as reference list caps, could effectively reduce citation inflation and stabilize bibliometric metrics?
• RQ5 How do confounding time-dependent variables like team size and citation counts interact with the disruption index, and what are the implications for causal inference?

Key findings

• The disruption index (CD) is systematically biased over time due to citation inflation, with its decline not reflecting reduced scientific disruption but structural changes in citation networks.
• The growth in reference list length (r_p) alone contributes significantly to citation inflation, with the total citation volume in the Web of Science network growing at an annual rate of 5.1% (g_C = g_n + g_r = 0.033 + 0.018).
• The disruption index converges to zero over time due to increased network density and triadic closure, especially from self-citations, which inflate the measure of consolidation (N_j) and distort disruption (CD_p).
• Empirical analysis reveals a positive relationship between CD_p and team size (k_p), contradicting prior reports of a negative relationship, indicating that time-dependent confounding invalidates causal claims.
• Computational simulations show that capping reference lists—especially with soft caps based on pages per article—can effectively reduce citation inflation and stabilize evaluation metrics.
• The authors release an ensemble of synthetic citation networks (DryadDisruption2023) to enable the testing of alternative, bias-resistant bibliometric indices.

Figure 2: Empirical analysis of the disruption index. (a) Schematic of the disruption index calculation based upon the sub-network revolving around the source publication/patent $p$ . The disruption index $CD_{p}$ can be calculated by identifying three non-overlapping subsets of $\{c\}_{p}=\{c\}_{i}