[Paper Review] A two-sided academic landscape: portrait of highly-cited documents in Google Scholar (1950-2013)

Alberto Martín‐Martín, Enrique Orduña‐Malea+2 more|arXiv (Cornell University)|Jul 11, 2016

scientometrics and bibliometrics research46 references3 citations

TL;DR

This study analyzes the top 1,000 most-cited documents per year (1950–2013) in Google Scholar, identifying their types, languages, availability, and versions. It reveals that highly cited works are predominantly English-language journal articles and books, with strong online PDF availability, offering a broader academic portrait than traditional databases due to Google Scholar’s extensive coverage of non-journal materials.

ABSTRACT

The main objective of this paper is to identify the set of highly-cited documents in Google Scholar and to define their core characteristics (document types, language, free availability, source providers, and number of versions), under the hypothesis that the wide coverage of this search engine may provide a different portrait about this document set respect to that offered by the traditional bibliographic databases. To do this, a query per year was carried out from 1950 to 2013 identifying the top 1,000 documents retrieved from Google Scholar and obtaining a final sample of 64,000 documents, of which 40% provided a free full-text link. The results obtained show that the average highly-cited document is a journal article or a book (62% of the top 1% most cited documents of the sample), written in English (92.5% of all documents) and available online in PDF format (86.0% of all documents). Yet, the existence of errors especially when detecting duplicates and linking cites properly must be pointed out. The fact of managing with highly cited papers, however, minimizes the effects of these limitations. Given the high presence of books, and to a lesser extend of other document types (such as proceedings or reports), the research concludes that Google Scholar data offer an original and different vision of the most influential academic documents (measured from the perspective of their citation count), a set composed not only by strictly scientific material (journal articles) but academic in its broad sense

Motivation & Objective

• To identify and characterize the set of highly-cited documents in Google Scholar across a 64-year period.
• To compare the composition of highly cited documents in Google Scholar with that found in traditional bibliographic databases.
• To assess the role of non-journal document types—such as books, conference proceedings, and reports—in academic influence.
• To evaluate the availability and accessibility of full-text versions, particularly free PDFs, in the Google Scholar index.
• To examine the impact of indexing limitations, such as duplicate detection and citation linking errors, on the analysis of highly cited works.

Proposed method

• A yearly query was conducted in Google Scholar from 1950 to 2013 to retrieve the top 1,000 most-cited documents per year.
• The final dataset comprised 64,000 unique documents after deduplication and quality filtering.
• Document characteristics were extracted, including type (e.g., journal article, book, conference paper), language, availability of free full-text links, source provider, and number of versions.
• Statistical analysis was applied to assess the distribution of document types, language prevalence, and digital availability across the sample.
• The study evaluated the reliability of Google Scholar’s citation indexing by assessing duplicate detection and citation linking accuracy.
• Findings were compared to traditional bibliographic databases to highlight differences in the representation of highly cited works.

Experimental results

Research questions

• RQ1 What types of documents dominate the set of highly cited works in Google Scholar compared to traditional databases?
• RQ2 What is the linguistic and format distribution (e.g., English, PDF) of the most-cited documents in Google Scholar?
• RQ3 How prevalent are free full-text versions among highly cited documents in Google Scholar?
• RQ4 To what extent do non-journal documents—such as books and reports—contribute to academic influence as measured by citation counts?
• RQ5 How do indexing errors in Google Scholar (e.g., duplicate detection, citation linking) affect the reliability of identifying highly cited works?

Key findings

• 62% of the top 1% most cited documents in the sample were journal articles or books, indicating their dominance in academic influence.
• 92.5% of all highly cited documents were published in English, highlighting the linguistic hegemony in high-impact scholarship.
• 86.0% of the documents were available in PDF format, reflecting strong digital accessibility through Google Scholar.
• 40% of the 64,000 documents in the sample provided a free full-text link, indicating significant open access availability.
• Books and other non-journal document types (e.g., conference proceedings, technical reports) were substantially overrepresented compared to traditional citation databases.
• Despite indexing errors in duplicate detection and citation linking, the high citation count of the documents mitigated the impact of these limitations on overall findings.