[Paper Review] Evaluation of the citation matching algorithms of CWTS and iFQ in comparison to Web of Science

Marlies Olensky, Marion Schmidt+1 more|arXiv (Cornell University)|Jul 12, 2015

Biomedical Text Mining and Ontologies21 citations

TL;DR

This study evaluates the citation matching algorithms of CWTS and iFQ against Web of Science (WoS) using a manually verified corpus, assessing how effectively each system corrects inaccuracies in cited references. CWTS's algorithm performs best (F1: 96.41%), followed closely by iFQ, while WoS shows notable deficits when references contain errors, highlighting the importance of robust matching in bibliometric research.

ABSTRACT

The results of bibliometric studies provided by bibliometric research groups, e.g. the Centre for Science and Technology Studies (CWTS) and the Institute for Research Information and Quality Assurance (iFQ), are often used in the process of research assessment. Their databases use Web of Science (WoS) citation data, which they match according to their own matching algorithms - in the case of CWTS for standard usage in their studies and in the case of iFQ on an experimental basis. Since the problem of non-matched citations in WoS persists because of inaccuracies in the references or inaccuracies introduced in the data extraction process, it is important to ascertain how well these inaccuracies are rectified in these citation matching algorithms. This paper evaluates the algorithms of CWTS and iFQ in comparison to WoS in a quantitative and a qualitative analysis. The analysis builds upon the methodology and the manually verified corpus of a previous study. The algorithm of CWTS performs best, closely followed by that of iFQ. The WoS algorithm still performs quite well (F1 score: 96.41 percent), but shows deficits in matching references containing inaccuracies. An additional problem is posed by incorrectly provided cited reference information in source articles by WoS.

Motivation & Objective

• To assess the performance of citation matching algorithms used by CWTS and iFQ in correcting inaccuracies present in cited references from Web of Science.
• To identify systematic weaknesses in Web of Science’s own citation matching process, particularly when source references contain errors.
• To compare the effectiveness of CWTS and iFQ algorithms against WoS in terms of precision, recall, and F1 score using a manually verified reference corpus.
• To evaluate how well each system handles real-world citation errors introduced during data extraction or reference entry.
• To provide empirical evidence on the reliability of bibliometric data derived from these systems in research assessment contexts.

Proposed method

• Utilized a manually verified corpus of citation pairs from a prior study to serve as the ground truth for evaluation.
• Applied the citation matching algorithms of CWTS, iFQ, and Web of Science to the same reference set to compare matching outcomes.
• Calculated standard information retrieval metrics—precision, recall, and F1 score—for each system’s performance.
• Conducted both quantitative analysis (using F1 scores) and qualitative assessment of matching errors and patterns.
• Focused on cases where cited references contained inaccuracies (e.g., missing or incorrect authors, titles, DOIs) to evaluate error correction capability.
• Used the same evaluation framework as a prior study to ensure consistency and comparability of results.

Experimental results

Research questions

• RQ1 How do the citation matching algorithms of CWTS and iFQ compare to Web of Science in terms of F1 score on a manually verified citation corpus?
• RQ2 To what extent can CWTS and iFQ correct citation inaccuracies that are present in Web of Science’s reference data?
• RQ3 What types of citation errors are most frequently misclassified or missed by each system’s algorithm?
• RQ4 Does Web of Science’s own matching algorithm perform reliably when faced with erroneous or incomplete cited reference information?
• RQ5 How do the performance characteristics of CWTS and iFQ differ in handling complex or ambiguous citation formats?

Key findings

• The CWTS citation matching algorithm achieved the highest F1 score of 96.41%, outperforming both iFQ and Web of Science.
• The iFQ algorithm performed closely behind CWTS, indicating strong error correction capability for citation inaccuracies.
• Web of Science’s own algorithm achieved a high but not perfect F1 score of 96.41%, but showed significant deficits when cited references contained errors.
• A key limitation of Web of Science was its inability to effectively correct inaccuracies introduced in the original source articles or during data extraction.
• The study confirmed that citation matching algorithms are critical for bibliometric accuracy, especially when source data contains errors.
• The results suggest that CWTS and iFQ provide more robust solutions than Web of Science for handling citation data with imperfections in research assessment contexts.