[Paper Review] Effects of Research Paper Promotion via ArXiv and X

Chhandak Bagchi, Eric Malmi+1 more|arXiv (Cornell University)|Jan 20, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This study analyzes the impact of early ArXiv preprint publication and X (formerly Twitter) promotion on citation counts in computer science, using a large dataset of 10 years of publications. It finds that early ArXiv submission boosts citations by 21.1±17.4, revisions by 18.4±17.6, and X promotion by 44.4±8 citations over five years, highlighting social media and preprint sharing as major drivers of scientific impact.

ABSTRACT

In the evolving landscape of scientific publishing, it is important to understand the drivers of high-impact research, to equip scientists with actionable strategies to enhance the reach of their work, and to understand trends in the use of modern scientific publishing tools to inform their further development. Here, we study trends in the use of early preprint publications and revisions on ArXiv and the use of X (formerly Twitter) for promotion of such papers in computer science and physics. We find that early submissions to ArXiv and promotion on X have soared in recent years. Estimating the effect that the use of each of these modern affordances has on the number of citations of scientific publications, we find that peer-reviewed conference papers in computer science that are submitted early to ArXiv gain on average $21.1 \pm 17.4$ more citations, revised on ArXiv gain $18.4 \pm 17.6$ more citations, and promoted on X gain $44.4 \pm 8$ more citations in the first 5 years from an initial publication. In contrast, journal articles in physics experience comparatively lower boosts in citation counts, with increases of $3.9 \pm 1.1$, $4.3 \pm 0.9$, and $6.9 \pm 3.5$ citations respectively for the same interventions. Our results show that promoting one's work on ArXiv or X has a large impact on the number of citations, as well as the number of influential citations computed by Semantic Scholar, and thereby on the career of researchers. These effects are present also for publications in physics, but they are relatively smaller. The larger relative effect sizes, effects of promotion accumulating over time, and elevated unpredictability of the number of citations in computer science than in physics suggest a greater role of world-of-mouth spreading in computer science than in physics.

Motivation & Objective

• To understand the rising trends in early preprint publication on ArXiv and promotion on X in computer science.
• To quantify the causal effect of ArXiv submission timing, revisions, and X promotion on citation counts.
• To assess whether modern impact metrics like Semantic Scholar's influential citations are similarly affected by social media exposure.
• To evaluate the implications of these trends for scientific evaluation, peer review, and future publishing systems.
• To explore whether socially-informed citation metrics can reduce bias from viral promotion.

Proposed method

• The study uses a large, cross-referenced dataset of computer science publications from 2013 to 2023, linking conference proceedings with ArXiv preprints and X (Twitter) activity.
• It applies doubly robust estimation to isolate causal effects of ArXiv submission timing, revisions, and X promotion on citation counts, controlling for confounding variables.
• Citation counts are measured over the first five years post-publication, using Semantic Scholar’s data for both total and influential citations.
• The analysis distinguishes between author-tweets and retweets by others to assess differential impact on citation outcomes.
• The study evaluates the volatility of modern impact metrics under social media influence and proposes network-normalized citation graphs to adjust for social bias.
• It classifies citations as topical or non-topical based on semantic similarity and author relationships to assess citation relevance.

Experimental results

Research questions

• RQ1 RQ1: Are early submissions to ArXiv and revisions, along with promotion on X, becoming standard practices in computer science?
• RQ2 RQ2: What is the effect of publishing a paper on ArXiv before conference proceedings on its citation count?
• RQ3 RQ3: What is the effect of revising a paper on ArXiv on its citation count?
• RQ4 RQ4.1: Can promoting a paper on X lead to more citations?
• RQ5 RQ5: Do newer, socially-informed impact metrics reduce the influence of social media on citation counts?

Key findings

• Early submission to ArXiv increases citation counts by 21.1±17.4 citations over five years compared to late submission.
• Revising a paper on ArXiv leads to an average increase of 18.4±17.6 additional citations in the first five years.
• Promoting a paper on X results in a significant boost of 44.4±8 more citations over five years, indicating a strong social media effect.
• Both total citations and Semantic Scholar’s influential citation counts are highly sensitive to promotion on ArXiv and X, suggesting social influence distorts traditional impact measures.
• The effect of author tweets is significantly higher than retweets by others, indicating that author-driven promotion is more impactful.
• Modern metrics like network-normalized citation counts may reduce social media bias, but no current metric is fully immune to gaming under Goodhart’s law.