[Paper Review] How journal rankings can suppress interdisciplinarity. The case of innovation studies in business and management

Ismael Ràfols, Loet Leydesdorff+3 more|arXiv (Cornell University)|May 6, 2011

scientometrics and bibliometrics research16 citations

TL;DR

This study uses citation and publication data to demonstrate that journal rankings systematically favor mono-disciplinary research by privileging journals with narrower disciplinary scope, thereby disadvantaging interdisciplinary Innovation Studies units. The key finding is that Business & Management schools receive more favorable performance assessments not due to higher quality, but due to the structural bias in journal rankings that undervalues interdisciplinary scholarship.

ABSTRACT

This study provides quantitative evidence on how the use of journal rankings can disadvantage interdisciplinary research in research evaluations. Using publication and citation data, it compares the degree of interdisciplinarity and the research performance of a number of Innovation Studies units with that of leading Business & Management schools in the UK. On the basis of various mappings and metrics, this study shows that: (i) Innovation Studies units are consistently more interdisciplinary in their research than Business & Management schools; (ii) the top journals in the Association of Business Schools' rankings span a less diverse set of disciplines than lower-ranked journals; (iii) this results in a more favourable assessment of the performance of Business & Management schools, which are more disciplinary-focused. This citation-based analysis challenges the journal ranking-based assessment. In short, the investigation illustrates how ostensibly 'excellence-based' journal rankings exhibit a systematic bias in favour of mono-disciplinary research. The paper concludes with a discussion of implications of these phenomena, in particular how the bias is likely to affect negatively the evaluation and associated financial resourcing of interdisciplinary research organisations, and may result in researchers becoming more compliant with disciplinary authority over time.

Motivation & Objective

• To investigate how journal rankings affect the evaluation of interdisciplinary research in academic performance assessments.
• To compare the interdisciplinarity and research performance of Innovation Studies units with Business & Management schools in the UK.
• To assess whether the journal ranking systems used in research evaluations systematically favor mono-disciplinary over interdisciplinary research.
• To examine the implications of such biases for funding allocation and institutional compliance with disciplinary norms.

Proposed method

• Quantitative analysis of publication and citation data from UK-based research units in Innovation Studies and Business & Management.
• Use of citation-based metrics to assess research performance across different disciplinary groupings.
• Mapping of journal disciplinary diversity using the journals cited by research units to evaluate the breadth of disciplinary engagement.
• Comparison of journal rankings (e.g., Association of Business Schools) with the actual disciplinary diversity of journals to identify bias.
• Application of metrics to compare interdisciplinarity levels between Innovation Studies and Business & Management units.
• Analysis of how ranking systems influence performance assessment outcomes despite differences in research scope and collaboration patterns.

Experimental results

Research questions

• RQ1 To what extent are Innovation Studies units more interdisciplinary than Business & Management schools in the UK?
• RQ2 How does the disciplinary diversity of top-ranked journals compare to that of lower-ranked journals in the Association of Business Schools ranking?
• RQ3 To what extent do journal rankings distort the assessment of research performance in favor of mono-disciplinary research?
• RQ4 What are the implications of these biases for the funding and institutional positioning of interdisciplinary research units?

Key findings

• Innovation Studies units consistently demonstrate higher levels of interdisciplinarity compared to Business & Management schools.
• Top-ranked journals in the Association of Business Schools' rankings exhibit less disciplinary diversity than lower-ranked journals.
• The citation-based performance assessment reveals that Business & Management schools receive more favorable evaluations due to the structural bias in journal rankings.
• Journal rankings systematically favor mono-disciplinary research by privileging journals with narrower disciplinary scopes.
• The bias in journal rankings leads to an unfair advantage for disciplinary-focused research, distorting performance evaluations.
• This systemic bias may discourage interdisciplinary research and pressure researchers to conform to disciplinary norms to secure funding and recognition.