[Paper Review] Citations Systematically Misrepresent the Quality and Impact of Research Articles: Survey and Experimental Evidence from Thousands of Citers

Misha Teplitskiy, Eamon Duede+2 more|arXiv (Cornell University)|Feb 24, 2020

scientometrics and bibliometrics research15 references9 citations

TL;DR

This study reveals that citation counts systematically distort perceptions of research quality and impact. Using a survey of 12,670 authors across 15 fields, it shows that citing behavior is biased: high citation counts lead to overestimation of quality in less-cited papers, while highly cited papers are underestimated in impact due to their frequent use for minor or zero-influence references.

ABSTRACT

Citations are ubiquitous in evaluating research, but how exactly they relate to what they are thought to measure (quality and intellectual impact) is unclear. We investigate the relationships between citations, quality, and impact using a survey with an embedded experiment in which 12,670 authors in 15 academic fields describe about 25K specific referencing decisions. Results suggest that citation counts, when equated with quality and impact, are biased in opposite directions. First, experimentally exposing papers' actual citation counts during the survey causes respondents to perceive all but the top 10% cited papers as of lower quality. Because perceptions of quality are a key factor in citing decisions, citation counts are likely to endogenously cause more citing of top papers and equating them with quality overestimates the actual quality of those papers. Conversely, 54% of references had either zero or minor influence on authors who cite them, but references to highly cited papers were about 200% more likely to denote substantial impact. Equating citations with impact thus underestimates the impact of highly cited papers. Real citation practices thus reveal that citations are biased measures of quality and impact.

Motivation & Objective

• To investigate how citation counts influence researchers' perceptions of research quality and intellectual impact.
• To examine the disconnect between citation counts and actual influence in citing decisions.
• To assess whether equating citations with quality or impact leads to systematic biases in research evaluation.
• To understand the role of citation visibility in shaping authors' referencing behavior and perceived value of cited works.

Proposed method

• Conducted a large-scale survey with 12,670 authors across 15 academic fields to collect data on referencing decisions.
• Embedded an experiment in the survey where participants were exposed to actual citation counts of cited papers.
• Collected data on perceived quality and influence of cited works, distinguishing between zero, minor, and substantial impact.
• Analyzed the relationship between citation counts and perceived quality/impact using multivariate regression models.
• Classified referencing decisions based on influence level (zero, minor, substantial) to assess impact bias.
• Used field-level controls to account for disciplinary differences in citation practices and evaluation norms.

Experimental results

Research questions

• RQ1 How do actual citation counts affect researchers' perceptions of the quality of cited papers?
• RQ2 To what extent do citations reflect substantial intellectual influence versus minor or no influence on citing works?
• RQ3 How does the visibility of citation counts bias evaluations of research quality and impact?
• RQ4 Why do highly cited papers appear to have lower perceived impact despite their high citation frequency?

Key findings

• Exposing actual citation counts during the survey led respondents to perceive all but the top 10% of cited papers as lower in quality, indicating a bias toward overestimating quality in highly cited works.
• 54% of all references had either zero or minor influence on citing authors, revealing that most citations do not contribute substantially to the citing work’s intellectual content.
• References to highly cited papers were approximately 200% more likely to denote substantial intellectual impact compared to references to less-cited papers.
• Citation counts systematically overestimate the quality of top-cited papers due to endogenous citation dynamics, where visibility drives further citing.
• Equating citations with impact underestimates the true influence of highly cited papers because many citations are not intellectually consequential.
• The study demonstrates that citation metrics are flawed proxies for both quality and impact, with biases operating in opposite directions depending on citation frequency.