[Paper Review] 2016 Google Scholar Metrics released: a matter of languages... and something else

Alberto Martín‐Martín, Juan Manuel Ayllón+2 more|arXiv (Cornell University)|Jan 1, 2016

Linguistics, Language Diversity, and Identity5 references3 citations

TL;DR

This paper critically evaluates the 2016 Google Scholar Metrics release, identifying persistent limitations such as inconsistent journal indexing, duplicate entries for multilingual journals, and the removal of working paper series like CEPR and IZA. It highlights the addition of five new language rankings (Russian, Korean, Polish, Ukrainian, Indonesian) and the exclusion of Italian and Dutch, while calling for structural improvements to enhance transparency, accuracy, and usability of the tool for scholarly evaluation.

ABSTRACT

The 2016 edition of Google Scholar Metrics was released on July 15th 2016. There haven't been any structural changes respect to previous versions, which means that most of its limitations still persist. The biggest changes are the addition of five new language rankings (Russian, Korean, Polish, Ukrainian, and Indonesian) and elimination of two other language rankings (Italian and Dutch). In addition, for reasons still unknown, this new edition doesn't include as many working paper and discussion paper series as previous editions.

Motivation & Objective

• To assess the 2016 Google Scholar Metrics release for structural and methodological consistency.
• To identify and document persistent limitations in journal indexing, citation data, and classification.
• To investigate the impact of language-specific rankings and the removal of certain language categories (Italian, Dutch).
• To examine the disappearance of major working paper series (e.g., CEPR, IZA) from the rankings.
• To propose actionable improvements for Google Scholar Metrics to enhance reliability, transparency, and usability in scholarly evaluation.

Proposed method

• Systematic comparison of Google Scholar Metrics 2016 with the 2015 version using visual and textual analysis of navigation, rankings, and search results.
• Identification and documentation of duplicate journal entries, particularly for multilingual publications such as Revista Española de Cardiología.
• Analysis of citation data inconsistencies, including incorrect links, dead links, and multiple source references for the same journal.
• Evaluation of indexing criteria, including the 100-article minimum requirement and its impact on non-English and local journals.
• Assessment of search functionality, including keyword-based retrieval and result limitations (e.g., 100 results per language, 20 per keyword).
• Proposal of 15 improvement suggestions based on bibliometric best practices, including data transparency, export functionality, and error reporting.

Experimental results

Research questions

• RQ1 What changes were introduced in the 2016 Google Scholar Metrics release compared to previous versions?
• RQ2 Why were CEPR and IZA discussion paper series removed from the 2016 rankings despite strong citation records?
• RQ3 How do indexing inconsistencies—such as duplicate entries or language-specific journal splits—affect the reliability of Google Scholar Metrics?
• RQ4 What are the implications of excluding certain language rankings (e.g., Italian, Dutch) while adding others (e.g., Korean, Russian)?
• RQ5 What structural and methodological improvements are needed to enhance the transparency and accuracy of Google Scholar Metrics as a scholarly evaluation tool?

Key findings

• The 2016 Google Scholar Metrics release introduced five new language rankings (Russian, Korean, Polish, Ukrainian, Indonesian) but removed Italian and Dutch, with no clear justification provided.
• Major working paper series such as CEPR Discussion Papers (h5-index: 112) and IZA Discussion Papers (h5-index: 82) were no longer listed, despite strong citation performance in the prior version.
• Persistent indexing errors were found, including duplicate entries for the same journal (e.g., Brazilian Journal of Anesthesiology with three entries) and inconsistent treatment of multilingual journals.
• The journal Revista Española de Cardiología was merged across language versions, while other multilingual journals (e.g., Revista Portuguesa de Neumologia) were not properly separated, leading to data contamination.
• Links to articles were frequently incorrect or led to unrelated publications, and some journals displayed multiple title variants due to data aggregation from disparate sources.
• The product lacks transparency in inclusion criteria, time frame selection, and classification rules, and does not allow users to view all cited or uncited articles, export data, or report errors.