[论文解读] Bibliometrics-based heuristics: What is their definition and how can they be studied?

Lutz Bornmann, Sven E. Hug|arXiv (Cornell University)|Oct 30, 2018

Experimental Behavioral Economics Studies参考文献 53被引用 4

一句话总结

本文提出了基于文献计量的启发式方法（BBHs），作为利用部分文献计量数据（如发表数量和引用次数）高效评估研究质量的简单、透明的决策规则。BBHs基于快速-节俭启发式理论，无需复杂模型即可实现准确、低成本的评估，为研究评估提供了一种科学依据充分的替代方案，避免了盲目引用计数的弊端。

ABSTRACT

When scientists study the phenomena they are interested in, they apply sound methods and base their work on theoretical considerations. In contrast, when the fruits of their research is being evaluated, basic scientific standards do not seem to matter. Instead, simplistic bibliometric indicators (i.e., publications and citation counts) are, paradoxically, both widely used and criticized without any methodological and theoretical framework that would serve to ground both use and critique. Yet, Bornmann and Marewski [1] proposed such a framework recently. They developed bibliometrics-based heuristics (BBHs) based on the fast-and-frugal heuristics approach [2] to decision making, in order to conceptually understand and empirically investigate the quantitative evaluation of research as well as to effectively train end-users of bibliometrics (e.g., science managers, scientists). Heuristics are decision strategies that use part of the available information and ignore the rest. By exploiting the statistical structure of task environments, they can aid to make accurate, fast, effortless, and cost-efficient decisions without that trade-offs are incurred. Because of their simplicity, heuristics are easy to understand and communicate, enhancing the transparency of decision processes. In this commentary, we explain several BBHs and discuss how such heuristics can be employed in practice (using the evaluation of applicants for funding programs as one example). Furthermore, we outline why heuristics can perform well, and how they and their fit to task environments can be studied. In pointing to the potential of research on BBHs and to the risks that come with an under-researched, mindless usage of bibliometrics, this commentary contributes to make research evaluation more scientific.

研究动机与目标

• 解决在广泛使用发表数量和引用次数等简单文献计量指标时，缺乏理论和方法论基础的问题。
• 为理解与培训科研评估中的终端用户（如科研管理者和研究人员）提供概念性与实证性框架。
• 展示启发式方法如何在忽略部分可用数据的情况下，仍能实现快速、准确且透明的科研评估决策。
• 为科研评估中使用和批判文献计量指标提供科学基础。
• 推动对BBHs的研究，以降低文献计量无序、盲目应用所带来的风险。

提出的方法

• 采用认知科学中的快速-节俭启发式框架，以建模在不确定性下的决策过程。
• 将BBHs定义为仅使用最少且相关文献计量信息（如发表数量、引用次数）进行决策的策略，同时忽略其他数据。
• 设计能利用科研评估环境统计结构的启发式方法，以提升决策的准确性和效率。
• 将BBHs应用于现实场景（如资助申请人的评估），以证明其实际可用性。
• 通过理论与实证分析，研究BBHs的性能及其与环境的匹配度。
• 强调透明性与沟通简便性，以增强科研评估情境中对BBHs的信任与采纳。

实验结果

研究问题

• RQ1 如何将文献计量指标概念化为启发式方法，而非刚性指标？
• RQ2 BBHs在准确性和透明度方面相较于传统引用计数法有何改进？
• RQ3 如何在不同评估环境中系统性地研究与验证BBHs？
• RQ4 BBHs在何种条件下表现良好，其与任务环境统计结构的匹配关系如何？
• RQ5 如何有效培训终端用户（如科研管理者）在实践中应用BBHs？

主要发现

• BBHs为科研评估中随意使用文献计量指标提供了理论基础坚实的替代方案。
• 通过聚焦少数关键数据点，BBHs可在不牺牲可靠性的前提下，实现快速、准确且成本低廉的决策。
• BBHs的性能取决于其与评估环境统计结构的匹配程度，从而增强了其鲁棒性。
• BBHs具有透明性且易于沟通，提升了评估过程的正当性与可信度。
• 实证研究表明，由于其简洁性与适应性，BBHs在现实环境中可能优于复杂模型。
• 该框架可实现对终端用户在科研评估中的系统性培训，降低文献计量数据被误用的风险。