[Paper Review] The weakening relationship between the Impact Factor and papers' citations in the digital age

George A. Lozano, Vincent Larivière+1 more|arXiv (Cornell University)|May 19, 2012

scientometrics and bibliometrics research17 references4 citations

TL;DR

This study analyzes the declining correlation between journal Impact Factors (IF) and individual paper citation counts from 1902 to 2009, showing that since 1990, the relationship has weakened due to the digital transition. The shift to online access allows papers to be cited based on merit rather than journal prestige, reducing the predictive power of IF for individual paper impact—especially in physics, which adopted digital formats earlier.

ABSTRACT

Historically, papers have been physically bound to the journal in which they were published but in the electronic age papers are available individually, no longer tied to their respective journals. Hence, papers now can be read and cited based on their own merits, independently of the journal's physical availability, reputation, or Impact Factor. We compare the strength of the relationship between journals' Impact Factors and the actual citations received by their respective papers from 1902 to 2009. Throughout most of the 20th century, papers' citation rates were increasingly linked to their respective journals' Impact Factors. However, since 1990, the advent of the digital age, the strength of the relation between Impact Factors and paper citations has been decreasing. This decrease began sooner in physics, a field that was quicker to make the transition into the electronic domain. Furthermore, since 1990, the proportion of highly cited papers coming from highly cited journals has been decreasing, and accordingly, the proportion of highly cited papers not coming from highly cited journals has also been increasing. Should this pattern continue, it might bring an end to the use of the Impact Factor as a way to evaluate the quality of journals, papers and researchers.

Motivation & Objective

• To investigate how the relationship between journal Impact Factors and individual paper citation counts has evolved over time.
• To assess the impact of digital publishing on the influence of journal reputation in shaping citation patterns.
• To determine whether the increasing availability of papers outside their original journals has weakened the predictive power of Impact Factors.
• To examine field-specific variations in this trend, particularly in physics versus other disciplines.
• To evaluate whether the continued decline in IF-paper citation correlation could undermine the future use of IF as a research evaluation metric.

Proposed method

• Analyzing citation data from 1902 to 2009 across multiple scientific disciplines.
• Measuring the correlation between journal Impact Factors and the actual citation counts of individual papers published in those journals.
• Comparing trends in citation patterns before and after 1990, marking the onset of widespread digital access.
• Tracking the proportion of highly cited papers originating from highly cited journals versus less prestigious journals over time.
• Using statistical analysis to detect shifts in the strength of the IF-citation relationship across fields, especially in physics.
• Employing longitudinal data to assess changes in the influence of journal-level metrics on individual paper impact.

Experimental results

Research questions

• RQ1 How has the correlation between journal Impact Factors and individual paper citation counts changed from 1902 to 2009?
• RQ2 When did the weakening of the IF-paper citation relationship begin, and what factors contributed to this shift?
• RQ3 How does the digital transition to online publishing affect the role of journal reputation in citation behavior?
• RQ4 To what extent does the proportion of highly cited papers from non-highly cited journals vary across disciplines?
• RQ5 Could the declining IF-paper citation correlation lead to the obsolescence of Impact Factors as a research evaluation tool?

Key findings

• The correlation between journal Impact Factors and individual paper citation counts has steadily weakened since 1990, indicating that journal prestige is less predictive of paper impact in the digital age.
• The decline in correlation began earlier in physics, a field that transitioned to digital publishing more rapidly than other disciplines.
• Since 1990, the proportion of highly cited papers originating from highly cited journals has decreased, while the share from less prestigious journals has increased.
• The trend suggests that papers are increasingly cited based on their intrinsic merit rather than the reputation of their publishing journal.
• The weakening relationship implies that Impact Factors may lose their utility as a proxy for individual paper or researcher quality if current trends continue.
• The findings challenge the continued use of Impact Factors as a primary metric for evaluating research quality, especially in fields with high digital accessibility.