[Paper Review] Bibliometrics-based heuristics: What is their definition and how can they be studied?

Lutz Bornmann, Sven E. Hug|arXiv (Cornell University)|Oct 30, 2018

Experimental Behavioral Economics Studies53 references4 citations

TL;DR

This paper introduces bibliometrics-based heuristics (BBHs) as simple, transparent decision rules that use partial bibliometric data—such as publication and citation counts—to evaluate research quality efficiently. Grounded in fast-and-frugal heuristics theory, BBHs enable accurate, low-cost evaluations in research assessment without requiring complex models, offering a scientifically grounded alternative to mindless citation counting.

ABSTRACT

When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research is being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publications and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Yet, Bornmann and Marewski [1] proposed such a framework recently. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach [2] to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.

Motivation & Objective

• To address the lack of theoretical and methodological grounding in the widespread use of simplistic bibliometric indicators like publication and citation counts.
• To propose a conceptual and empirical framework for understanding and training end-users in research evaluation, such as science managers and researchers.
• To demonstrate how heuristics can provide fast, accurate, and transparent decisions in research evaluation despite ignoring part of the available data.
• To establish a scientific basis for both the use and critique of bibliometric indicators in research assessment.
• To promote the study of BBHs as a way to reduce risks associated with unstructured, mindless application of bibliometrics.

Proposed method

• Adopting the fast-and-frugal heuristics framework from cognitive science to model decision-making under uncertainty.
• Defining BBHs as decision strategies that use minimal, relevant bibliometric information (e.g., number of publications, citations) while ignoring other data.
• Designing heuristics that exploit the statistical structure of research evaluation environments to improve decision accuracy and efficiency.
• Applying BBHs to real-world scenarios, such as evaluating funding applicants, to demonstrate practical usability.
• Using theoretical and empirical analysis to study the performance and environmental fit of BBHs.
• Emphasizing transparency and ease of communication to enhance trust and adoption in research evaluation contexts.

Experimental results

Research questions

• RQ1 How can bibliometric indicators be conceptualized as heuristics rather than rigid metrics?
• RQ2 In what ways do BBHs improve the accuracy and transparency of research evaluation compared to traditional citation counting?
• RQ3 How can BBHs be systematically studied and validated in different evaluation environments?
• RQ4 What are the conditions under which BBHs perform well, and how do they align with the statistical structure of task environments?
• RQ5 How can end-users such as science managers be effectively trained to apply BBHs in practice?

Key findings

• BBHs provide a theoretically grounded alternative to the arbitrary use of bibliometric indicators in research evaluation.
• By focusing on a few key data points, BBHs enable fast, accurate, and cost-efficient decisions without sacrificing reliability.
• The performance of BBHs depends on their alignment with the statistical structure of the evaluation environment, enhancing their robustness.
• BBHs are transparent and easy to communicate, which increases the legitimacy and trustworthiness of evaluation processes.
• Empirical studies show that BBHs can outperform complex models in real-world settings due to their simplicity and adaptability.
• The framework enables systematic training of end-users in research evaluation, reducing the risk of misuse of bibliometric data.