[Paper Review] Exploring the Effects of Data Set Choice on Measuring International Research Collaboration: an Example Using the ACM Digital Library and Microsoft Academic Graph

Ba Xuan Nguyen, Markus Luczak–Roesch+1 more|arXiv (Cornell University)|May 30, 2019

scientometrics and bibliometrics research5 references4 citations

TL;DR

This study investigates how different academic data sets—specifically the ACM Digital Library and Microsoft Academic Graph—affect measurements of international research collaboration (IRC). By comparing IRC metrics across both sources, the authors demonstrate significant discrepancies in collaboration rates and country rankings, highlighting that data set choice critically influences research evaluation outcomes in scientometrics.

ABSTRACT

International research collaboration (IRC) measurement is important because countries can and want to benefit from international collaboration but performing the same measurement procedure on different data sets can lead to different results. This study aims to explore the effects of data set choice on IRC measurement.

Motivation & Objective

• To examine how the choice of academic data set affects the measurement of international research collaboration (IRC).
• To compare IRC metrics derived from the ACM Digital Library and Microsoft Academic Graph across multiple dimensions.
• To identify systematic differences in collaboration rates, country rankings, and author coverage between the two data sets.
• To provide empirical evidence that data set selection can lead to substantially different conclusions in scientometric studies.
• To inform researchers and policymakers about the importance of data source selection in evaluating international research collaboration.

Proposed method

• The study extracts publication records from the ACM Digital Library and Microsoft Academic Graph for a defined time period.
• It identifies co-authorship relationships and assigns countries to authors based on institutional affiliations.
• IRC is measured as the proportion of publications with at least one co-author from a different country.
• The authors compare IRC rates, country-level collaboration rankings, and author coverage between the two data sets.
• Statistical comparisons and correlation analyses are used to assess the consistency and divergence of results.
• The analysis includes sensitivity checks on data quality and completeness across sources.

Experimental results

Research questions

• RQ1 How do IRC rates differ when measured using the ACM Digital Library versus Microsoft Academic Graph?
• RQ2 To what extent do country-level collaboration rankings vary between the two data sets?
• RQ3 How does the coverage of authors and publications differ between the ACM Digital Library and Microsoft Academic Graph?
• RQ4 What are the implications of data set choice for the reliability and validity of IRC measurements?
• RQ5 Are there systematic biases in one data set that affect the measurement of international collaboration?

Key findings

• The ACM Digital Library reports a higher international collaboration rate (38.6%) compared to Microsoft Academic Graph (28.7%) for the same publication period.
• Country-level collaboration rankings show significant divergence, with some nations ranking much higher in one data set than the other.
• The ACM Digital Library covers a larger proportion of authors and publications in computer science, particularly from non-US institutions.
• Microsoft Academic Graph exhibits lower coverage of non-English and non-US publications, affecting IRC measurement accuracy.
• The study identifies that data set selection can lead to different conclusions about the extent and structure of international research collaboration.
• Discrepancies are attributed to differences in data collection, indexing policies, and institutional affiliation extraction methods between the two sources.