[Paper Review] The Operationalization of "Fields" as WoS Subject Categories (WCs) in Evaluative Bibliometrics: The cases of "Library and Information Science" and "Science & Technology Studies"

Loet Leydesdorff, Lutz Bornmann|arXiv (Cornell University)|Jul 29, 2014

scientometrics and bibliometrics research47 references12 citations

TL;DR

This paper critiques the use of Web of Science Subject Categories (WCs) as reference sets for bibliometric normalization in evaluative research, demonstrating that indexer effects and inconsistent categorization undermine reliability in fields like Library and Information Science and Science & Technology Studies. It argues for replacing 'best practices' with 'best possible practices' through improved classification methods, such as algorithmic clustering or professional indexing terms, to enhance evaluation accuracy and validity.

ABSTRACT

Normalization of citation scores using reference sets based on Web-of-Science Subject Categories (WCs) has become an established ("best") practice in evaluative bibliometrics. For example, the Times Higher Education World University Rankings are, among other things, based on this operationalization. However, WCs were developed decades ago for the purpose of information retrieval and evolved incrementally with the database; the classification is machine-based and partially manually corrected. Using the WC "information science & library science" and the WCs attributed to journals in the field of "science and technology studies," we show that WCs do not provide sufficient analytical clarity to carry bibliometric normalization in evaluation practices because of "indexer effects." Can the compliance with "best practices" be replaced with an ambition to develop "best possible practices"? New research questions can then be envisaged.

Motivation & Objective

• To examine the reliability of Web of Science Subject Categories (WCs) as reference sets for bibliometric normalization in evaluative research.
• To identify systematic flaws in WC-based normalization, particularly due to indexer effects and inconsistent categorization.
• To challenge the assumption that WCs provide a stable, discipline-specific basis for comparing citation impact across institutions or individuals.
• To advocate for alternative, more robust methods—such as algorithmic clustering or professional indexing terms—for defining reference sets in bibliometric evaluation.
• To stimulate a shift from established 'best practices' to more rigorous, research-driven 'best possible practices' in evaluative bibliometrics.

Proposed method

• Analyzes the operationalization of WCs in the Web of Science database, focusing on their historical development as retrieval tools rather than evaluation standards.
• Compares the classification of journals in 'Library and Information Science' and 'Science & Technology Studies' (STS) to assess consistency and coherence of reference sets.
• Evaluates the impact of multiple, overlapping, or ambiguous WCs on normalization outcomes, particularly in interdisciplinary fields.
• Reviews existing normalization methods, including CPP/Fcsm and MNCS, and highlights limitations in their reliance on WC-based journal categorization.
• Proposes alternative reference set generation methods, such as algorithmic clustering of citation relations (as in the Leiden Rankings) and professional indexing terms (e.g., MeSH, Chemical Abstracts).
• Discusses the trade-offs of each method, including issues of replicability, validation, and the 'double citation window' problem in citation-based normalization.

Experimental results

Research questions

• RQ1 To what extent do Web of Science Subject Categories (WCs) provide a stable and analytically coherent basis for bibliometric normalization in evaluative research?
• RQ2 How do indexer effects and inconsistent categorization of journals within WCs compromise the validity of citation normalization in fields like Library and Information Science and Science & Technology Studies?
• RQ3 Why is the current reliance on WCs—originally designed for information retrieval—problematic for modern evaluative bibliometrics?
• RQ4 What are the limitations of algorithmically generated fields (e.g., in the Leiden Rankings) in terms of interpretability, validation, and replicability?
• RQ5 Can alternative reference set definitions—such as professional indexing terms or citation-based clustering—offer more reliable and defensible normalization than WC-based approaches?

Key findings

• The Web of Science Subject Categories (WCs) were not designed for evaluative bibliometrics and exhibit significant indexer effects, undermining their reliability as reference sets for normalization.
• The field of 'Library and Information Science' is inconsistently categorized under the single WC 'information science & library science,' which may not reflect disciplinary boundaries or citation patterns accurately.
• Science and Technology Studies (STS) is not assigned a dedicated WC, leading to its dispersion across multiple categories and potential misclassification in normalization processes.
• The use of multiple, overlapping WCs for journals introduces ambiguity and reduces the analytical clarity of reference sets, especially in interdisciplinary research areas.
• Algorithmic clustering methods, such as those used in the Leiden Rankings, offer a more dynamic and citation-based alternative but face challenges in validation, naming, and replicability.
• The paper concludes that current 'best practices' in bibliometric normalization based on WCs are insufficient and that a shift toward 'best possible practices'—grounded in better classification systems—is both necessary and feasible.