[Paper Review] An analysis of the suitability of OpenAlex for bibliometric analyses

Juan Pablo Alperín, Jason Portenoy+3 more|arXiv (Cornell University)|Apr 26, 2024

Big Data and Business Intelligence15 citations

TL;DR

The study compares OpenAlex to Scopus, finding OpenAlex is a superset of Scopus and suitable for some analyses (notably country-level), but metadata accuracy and completeness require further research to broaden its reliable use.

ABSTRACT

Scopus and the Web of Science have been the foundation for research in the science of science even though these traditional databases systematically underrepresent certain disciplines and world regions. In response, new inclusive databases, notably OpenAlex, have emerged. While many studies have begun using OpenAlex as a data source, few critically assess its limitations. This study, conducted in collaboration with the OpenAlex team, addresses this gap by comparing OpenAlex to Scopus across a number of dimensions. The analysis concludes that OpenAlex is a superset of Scopus and can be a reliable alternative for some analyses, particularly at the country level. Despite this, issues of metadata accuracy and completeness show that additional research is needed to fully comprehend and address OpenAlex's limitations. Doing so will be necessary to confidently use OpenAlex across a wider set of analyses, including those that are not at all possible with more constrained databases.

Motivation & Objective

• Motivate the need for inclusive bibliometric data sources beyond Scopus and Web of Science.
• Evaluate OpenAlex as a data source for bibliometric analyses by directly comparing it with Scopus across several dimensions.
• Identify limitations in OpenAlex related to metadata accuracy and completeness.
• Assess the reliability of OpenAlex for country-level analyses and its potential for broader analyses.
• Suggest areas where further research is needed to improve OpenAlex's suitability.

Proposed method

• Collaborate with the OpenAlex team to perform a comparative analysis against Scopus across multiple dimensions.
• Assess dataset characteristics such as coverage, accuracy, and completeness.
• Evaluate the reliability of OpenAlex for country-level analyses specifically.
• Synthesize findings to determine scenarios where OpenAlex is a viable alternative.
• Discuss limitations and directions for future research to address metadata issues.

Experimental results

Research questions

• RQ1 Is OpenAlex a superset of Scopus in terms of coverage and inclusivity?
• RQ2 For which bibliometric analyses is OpenAlex a reliable alternative to Scopus or Web of Science?
• RQ3 What are the limitations in metadata accuracy and completeness for OpenAlex?
• RQ4 How does OpenAlex perform for country-level analyses compared with Scopus?
• RQ5 What further research is needed to address OpenAlex's limitations for broader analyses?

Key findings

• OpenAlex is a superset of Scopus.
• OpenAlex can be a reliable alternative for some analyses, particularly at the country level.
• Metadata accuracy and completeness in OpenAlex present limitations that require caution.
• Additional research is needed to understand and address OpenAlex's limitations.
• A broader range of analyses beyond those feasible with more constrained databases remains contingent on improving metadata quality.