[Paper Review] Google Scholar is manipulatable

Hazem Ibrahim, Fengyuan Liu+2 more|arXiv (Cornell University)|Feb 7, 2024

Artificial Intelligence in Healthcare and Education11 citations

TL;DR

The study shows Google Scholar can be manipulated via citation purchasing and fake profiles, providing evidence that citation counts can be bought and misleading in evaluative contexts.

ABSTRACT

Citations are widely considered in scientists' evaluation. As such, scientists may be incentivized to inflate their citation counts. While previous literature has examined self-citations and citation cartels, it remains unclear whether scientists can purchase citations. Here, we compile a dataset of ~1.6 million profiles on Google Scholar to examine instances of citation fraud on the platform. We survey faculty at highly-ranked universities, and confirm that Google Scholar is widely used when evaluating scientists. Intrigued by a citation-boosting service that we unravelled during our investigation, we contacted the service while undercover as a fictional author, and managed to purchase 50 citations. These findings provide conclusive evidence that citations can be bought in bulk, and highlight the need to look beyond citation counts.

Motivation & Objective

• Assess how widely researchers rely on citation metrics in recruitment and promotion decisions.
• Quantify Google Scholar's prominence as a source of citation data among academics at top universities.
• Identify patterns in suspicious Google Scholar profiles and potential manipulation techniques.
• Demonstrate the feasibility of purchasing citations and its impact on measured metrics.
• Propose a metric to flag potentially suspicious citation activity (the c2-index).

Proposed method

• Survey faculty at the top 10 universities to determine sources used for citation data.
• Curate a dataset of over 1.6 million Google Scholar profiles to analyze irregular citation patterns.
• Undercover experiment by purchasing 50 citations for a fictional author to demonstrate feasibility.
• Create a fictional Google Scholar profile and upload AI-generated articles to test moderation and indexing.
• Purchase citations via a third-party service and analyze the citing papers for evidence of bulk manipulation.
• Introduce the c2-index (citation concentration index) to flag highly concentrated bulk citations.

Figure 1: Survey responses from faculty of the top-10 ranked universities around the world. A , The percentage of faculty who consider citations when evaluating candidates (blue) and those who do not (red). B , Solid bars indicate, out of those who self-report considering citations when evaluating c

Experimental results

Research questions

• RQ1 What is the prevalence of using Google Scholar as a primary source for citation metrics among faculty evaluating candidates?
• RQ2 Can Google Scholar profiles be manipulated through citation purchasing or other means?
• RQ3 To what extent can citations be bought and how do such purchases manifest in citation patterns across papers?
• RQ4 Can a simple metric (c2-index) help identify suspicious citation behavior on Google Scholar?
• RQ5 What are the moderation and indexing vulnerabilities of Google Scholar when confronted with artificially generated content?

Key findings

• Google Scholar is the most popular source for citation metrics among evaluators, used by over 60% of respondents who consider citations.
• A proof-of-concept shows that 50 citations can be purchased for a fictional author within weeks, demonstrating bulk citation manipulation is feasible.
• Suspicious profiles exhibit sudden, peak-year citation spikes, high concentration of citations from a small set of papers, and heavy use of non-traditional sources (pre-prints).
• Compared to Scopus, suspicious authors show a large drop in cited counts (on average 96% vs 43%), indicating discrepancies across databases.
• The c2-index identifies profiles with unusually high concentration of citations from many papers; adjusted c2-index highlights potential manipulation risk.
• Citations can persist in Google Scholar even after source papers are removed, suggesting indexing without robust moderation.

Figure 2: A comparative analysis of suspicious authors and their matches. In each plot, red lines and red dots denote suspicious authors, while blue ones denote their matches. A , For the 4 years leading up to an author’s peak citations, the annual number of citations relative to the peak. B , Discr