[Paper Review] Evaluation of research and evolution of science indicators

Loet Leydesdorff|ArXiv.org|Nov 22, 2009

scientometrics and bibliometrics research37 references20 citations

TL;DR

This paper examines how the evolution of science indicators has been reshaped by systemic changes in academia, government, and industry relationships, particularly in the context of a knowledge-based economy and globalization. It argues that research evaluation has become embedded in complex, multidimensional communication systems, necessitating new approaches to scientometric measurement that reflect dynamic, globalized knowledge flows and shifting definitions of research.

ABSTRACT

Whereas the research program of the measurement of scientific communications emerged in a context where the delineations among academia, government, and industry were institutionalized, the systemic development of these relations during the second half of the 20th century has changed the system of reference for the evaluation of research. In a knowledge-based economy science fulfills functions that change the definitions of what is considered research and globalization has changed the relevance of national systems of reference. Science, of course, has been internationally oriented from its very beginning, but the entrainment of the research process in these global developments is reflected in the research evaluation and the scientometric measurement. In other words, the systems under study have become more complex. A complex dynamics can analytically be decomposed in several subdynamics. The evolving systems and subsystems communicate in different dimensions and the evaluation has become part of the codification of these communications.

Motivation & Objective

• To analyze how institutional changes in academia, government, and industry have redefined the systems of reference for research evaluation.
• To investigate the impact of globalization and the knowledge-based economy on the conceptualization and measurement of scientific research.
• To examine how the increasing complexity of research systems necessitates new approaches to scientometric indicators.
• To explore the role of communication dynamics in shaping the codification and evaluation of scientific output.
• To reassess the relevance and evolution of traditional science indicators in light of systemic shifts in research systems.

Proposed method

• Analytical decomposition of complex research systems into subdynamics to understand their interactions.
• Use of scientometric indicators as tools to measure and evaluate scientific communication and knowledge flows.
• Application of systems theory to model the evolving relationships between academia, government, industry, and global networks.
• Examination of historical developments in research evaluation from institutionalized boundaries to integrated, globalized systems.
• Integration of theoretical frameworks from physics and sociology of science to interpret changes in research measurement.
• Use of the journal article 'Current Science' as a case study for evaluating indicator evolution over time.

Experimental results

Research questions

• RQ1 How have institutional boundaries between academia, government, and industry evolved, and what impact has this had on research evaluation?
• RQ2 In what ways has globalization altered the relevance of national systems of reference in evaluating scientific research?
• RQ3 How has the shift to a knowledge-based economy redefined the criteria and functions of what is considered research?
• RQ4 What are the implications of complex, multidimensional communication systems for the development of science indicators?
• RQ5 How has the codification of scientific communication become integral to the evaluation process itself?

Key findings

• The institutional delineation between academia, government, and industry has eroded, leading to new systemic configurations that challenge traditional evaluation frameworks.
• Globalization has diminished the primacy of national systems in research evaluation, shifting focus toward transnational and interdisciplinary knowledge networks.
• The knowledge-based economy has expanded the functions of science beyond pure research to include innovation, policy, and economic development, altering the definition of 'research'.
• Scientific communication has become increasingly embedded in complex, dynamic systems where evaluation is no longer external but part of the codification process.
• Traditional science indicators are no longer sufficient to capture the multidimensional and evolving nature of contemporary research systems.
• The paper concludes that science indicators must evolve to reflect systemic complexity, global integration, and the changing roles of science in society.