[Paper Review] Disruptive papers in science are losing impact

An Zeng, Ying Fan+3 more|arXiv (Cornell University)|May 5, 2023

scientometrics and bibliometrics researchDecision Sciences3 citations

TL;DR

This study analyzes the evolving relationship between scientific impact (citations) and originality (disruption) over 70 years, finding a significant shift from positive to negative correlation. Using citation and disruption metrics across six disciplines, it reveals that highly disruptive papers now attract fewer citations, suggesting declining attention to radical innovation due to information overload and citation-driven impact metrics.

ABSTRACT

The impact and originality are two critical dimensions for evaluating scientific publications, measured by citation and disruption metrics respectively. Despite the extensive effort made to understand the statistical properties and evolution of each of these metrics, the relations between the two remain unclear. In this paper, we study the evolution during last 70 years of the correlation between scientific papers' citation and disruption, finding surprisingly a decreasing trend from positive to negative correlations over the years. Consequently, during the years, there are fewer and fewer disruptive works among the highly cited papers. These results suggest that highly disruptive studies nowadays attract less attention from the scientific community. The analysis on papers' references supports this trend, showing that papers citing older references, less popular references and diverse references become to have less citations. Possible explanations for the less attention phenomenon could be due to the increasing information overload in science, and citations become more and more prominent for impact. This is supported by the evidence that research fields with more papers have a more negative correlation between citation and disruption. Finally, we show the generality of our findings by analyzing and comparing six disciplines.

Motivation & Objective

• To investigate the temporal evolution of the relationship between scientific impact (citations) and originality (disruption) in research papers.
• To determine whether highly disruptive papers are still receiving high attention from the scientific community over time.
• To explore the role of reference patterns—such as age, popularity, and diversity—as proxies for originality and their influence on citation impact.
• To assess whether the negative correlation between disruption and citations is a general trend across diverse scientific fields.

Proposed method

• Measured paper impact using $C_5$, the number of citations received within 5 years of publication, to ensure temporal comparability.
• Calculated disruption using $D_5$, a metric based on the proportion of citing papers that do not cite the focal paper’s references, ranging from -1 to 1.
• Used reference diversity as an alternative originality measure, defined as the average of $1/(1 + n_{ij})$ over all reference pairs, where $n_{ij}$ is their historical co-citation count.
• Applied bootstrap resampling to test the statistical significance of differences in mean $C_5$ between highly disruptive papers and all papers in a given year.
• Analyzed data from six disciplines: physics, computer science, chemistry, biology, social science, and multidisciplinary science, using publicly available datasets from APS, ACS, SAGE, Cell, Microsoft Academic Graph, and AMiner.
• Examined the correlation between disruption and citations over time, and assessed how field-level publication volume influences this correlation.

Experimental results

Research questions

• RQ1 How has the correlation between citation impact and disruption evolved over the past 70 years in scientific literature?
• RQ2 Are highly disruptive papers still receiving high citation counts, or has their visibility declined over time?
• RQ3 To what extent do reference age, reference popularity, and reference diversity predict a paper’s citation impact?
• RQ4 Does the negative correlation between disruption and citations increase with the volume of scientific output in a given field?
• RQ5 Is the trend of declining impact for disruptive works consistent across multiple scientific disciplines?

Key findings

• The correlation between citation count ($C_5$) and disruption ($D_5$) has shifted from positive to negative over the past 70 years, indicating that disruptive papers are increasingly less cited.
• Highly disruptive papers now receive significantly fewer citations than average papers in the same year, with this gap growing over time.
• Papers citing older references, less popular references, and more diverse references tend to accumulate fewer citations, suggesting originality is penalized in citation-based evaluation.
• Research fields with higher publication volumes exhibit stronger negative correlations between disruption and citations, indicating field-level information overload intensifies the trend.
• The negative correlation between disruption and citations is consistently observed across six disciplines—physics, computer science, chemistry, biology, social science, and multidisciplinary science—demonstrating the generality of the finding.
• The decline in impact of disruptive works is likely driven by information overload and the growing dominance of citation counts in evaluating scientific impact.