[Paper Review] Reflexive Metrics: Reactivity and Practices of the Evaluation Culture in Astronomy

Julia Heuritsch|arXiv (Cornell University)|Dec 16, 2021

scientometrics and bibliometrics research6 citations

TL;DR

This paper applies a rational choice framework to analyze how evaluation metrics create an 'evaluation gap' in astronomy, where institutional pressures to meet bibliometric indicators conflict with intrinsic scientific values. It reveals that astronomers engage in 'gaming' strategies—like salami slicing and prioritizing easy publications—to comply with performance metrics, ultimately compromising research quality despite rational self-interest.

ABSTRACT

Diese Dissertation aus der Sparte der Reflexiven Bibliometrie erforscht die Rückwirkungen der Verwendung von quantitativen Indikatoren in der Wissenschaftsevaluation auf die Wissensproduktion und Forschungsqualität in der Astronomie. Eine qualitative Analyse der strukturellen Bedingungen der akamdeischen Astronomie anhand des Rational Choice Frameworks führen zur Beobachtung eines "Evaluation Gap" zwischen dem, was Indikatoren messen und dem, was Forscher unter Forschungsqualität verstehen. Die Analyse offenbart weiters einen Balanceakt, in dem Astronomen zwischen Publikationsdruck und Forschungsintegrität Kompromisse finden. Weiterführende quantitative Untersuchungen unter Einbezug von Organisational Culture Theories und Self-Determination Theory zeigen, dass kontrollierte Formen von Motivation zu einem erhöhten Publikationsdruck und wissenschaftlichem Fehlverhalten führen, während autonome Formen von Motivation das Gegenteil bewirken. Schließlich skizziert die Arbeit Wege zur Transformation der Forschungskultur hin zu mehr Vielfalt und Partizipation, einschließlich der Einführung offener Wissensmanagement-Infrastrukturen und kontinuierlicher, reflexiver Evaluationsprozesse.

Motivation & Objective

• To investigate the tension between intrinsic scientific values and extrinsic evaluation metrics in astronomy.
• To examine how institutional norms and forms of capital (funding, publication rates, telescope time) shape research behavior.
• To understand the mechanisms behind the 'evaluation gap' in observational astronomy.
• To explore how astronomers navigate competing motivations using rational, strategic behaviors.

Proposed method

• Applied a rational choice framework to interpret astronomers' strategic behaviors in response to evaluation systems.
• Conducted 19 semi-structured interviews with international observational astronomers.
• Analyzed institutional norms, material resources, and forms of capital (economic, cultural, social, symbolic) in the context of research evaluation.
• Used qualitative thematic analysis to identify patterns in how researchers balance intrinsic motivation with performance demands.
• Reconstructed the moral economy of astronomy through institutional and cultural analysis.
• Integrated findings into a causal theory explaining how indicator use degrades research quality.

Experimental results

Research questions

• RQ1 How do astronomers reconcile intrinsic motivations for scientific discovery with extrinsic pressures to produce measurable outputs?
• RQ2 What institutional norms and forms of capital mediate the relationship between research quality and bibliometric performance?
• RQ3 How do astronomers strategically respond to evaluation systems in ways that may compromise research quality?
• RQ4 In what ways do performance indicators distort research practices in observational astronomy?
• RQ5 What structural conditions lead to a focus on publication quantity over quality?

Key findings

• Astronomers experience anomie, balancing intrinsic motivation to advance knowledge with extrinsic demands to meet performance indicators.
• Researchers employ 'gaming' strategies such as salami slicing and targeting low-hanging fruit to appear compliant with evaluation metrics.
• The evaluation system incentivizes publication quantity over research quality, leading to a systemic decline in scientific rigor.
• Institutional norms and access to capital (e.g., telescope time, funding) privilege certain institutions and individuals, reinforcing inequities.
• The current system functions as a 'naïve meritocracy' that masks power imbalances and political decision-making in resource allocation.
• The moral economy of astronomy is shaped by negotiated compromises between competing values, with performance indicators playing a central but distorting role.