[Paper Review] Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic

Martijn S. Visser, Nees Jan van Eck+1 more|arXiv (Cornell University)|May 21, 2020

scientometrics and bibliometrics research20 citations

TL;DR

This study conducts a large-scale, systematic comparison of five major bibliographic databases—Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic—across coverage, citation completeness, and accuracy from 2008 to 2017. It reveals significant disparities in document coverage and citation reliability, emphasizing the need to combine broad literature coverage with flexible filtering for robust bibliometric analysis.

ABSTRACT

We present a large-scale comparison of five multidisciplinary bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic. The comparison considers scientific documents from the period 2008-2017 covered by these data sources. Scopus is compared in a pairwise manner with each of the other data sources. We first analyze differences between the data sources in the coverage of documents, focusing for instance on differences over time, differences per document type, and differences per discipline. We then study differences in the completeness and accuracy of citation links. Based on our analysis, we discuss strengths and weaknesses of the different data sources. We emphasize the importance of combining a comprehensive coverage of the scientific literature with a flexible set of filters for making selections of the literature.

Motivation & Objective

• To evaluate and compare the coverage, completeness, and accuracy of five major multidisciplinary bibliographic databases.
• To identify systematic differences in document coverage across time, document types, and academic disciplines.
• To assess the reliability and consistency of citation links across the five data sources.
• To provide evidence-based guidance on selecting appropriate bibliographic databases for large-scale literature analysis.
• To highlight the importance of combining comprehensive coverage with flexible filtering in bibliometric research.

Proposed method

• Conducted a large-scale, pairwise comparison of Scopus against each of the other four data sources (Web of Science, Dimensions, Crossref, Microsoft Academic).
• Analyzed document coverage across the period 2008–2017, stratified by year, document type, and academic discipline.
• Evaluated citation link completeness and accuracy by comparing forward and backward citation records across sources.
• Used statistical and network-based methods to assess alignment and discrepancies in citation data.
• Applied standardized metrics to quantify overlap and divergence in document and citation records.
• Focused on multidisciplinary scientific literature to ensure generalizability across research fields.

Experimental results

Research questions

• RQ1 How do the five bibliographic data sources compare in terms of coverage of scientific documents from 2008 to 2017?
• RQ2 What are the differences in coverage across document types (e.g., articles, reviews, conference papers) and academic disciplines?
• RQ3 How consistent are citation links between the data sources, and which sources exhibit higher completeness and accuracy?
• RQ4 What are the strengths and weaknesses of each data source in supporting large-scale bibliometric research?
• RQ5 To what extent can combining multiple sources improve the reliability and comprehensiveness of literature analysis?

Key findings

• Scopus demonstrated broader coverage than Web of Science, especially in STEM and health sciences, with higher document counts across most years.
• Dimensions showed strong coverage in clinical medicine and life sciences, often outperforming Web of Science and Crossref in these domains.
• Crossref had the highest proportion of missing or incomplete citation links, particularly for older publications and non-English journals.
• Microsoft Academic showed notable gaps in coverage for non-English and non-English-language journals, especially before 2010.
• Web of Science had the most consistent citation data but the narrowest coverage, particularly in engineering and computer science.
• The study found that no single source provides complete coverage; combining multiple sources significantly improves retrieval completeness and accuracy.