[Paper Review] Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton

Pablo Dorta–González, Alejandro Rodrı́guez+1 more|arXiv (Cornell University)|Mar 11, 2024

Social Capital and Networks4 citations

TL;DR

This large-scale study analyzes how scientific research influences policymaking using Altmetric and Overton data across nearly 125,000 policy journal articles. It finds that news mentions, social media engagement, and non-payed access significantly increase policy citations, while paywalled and fully open access articles are less likely to be cited in policy documents, highlighting the critical role of science communication in bridging research and policy.

ABSTRACT

This study investigates how scientific research influences policymaking by analyzing citations of research articles in policy documents (policy impact) for nearly 125,000 articles across 434 public policy journals. We reveal distinct citation patterns between policymakers and other stakeholders like researchers, journalists, and the public. News and blog mentions, social media engagement, and open access publications (excluding fully open access) significantly increase the likelihood of a research article being cited in policy documents. Conversely, articles locked behind paywalls and those published under the full open access model (based on Altmetric data) have a lower chance of being policy-cited. Publication year and policy type show no significant influence. Our findings emphasize the crucial role of science communication channels like news media and social media in bridging the gap between research and policy. Interestingly, academic citations hold a weaker influence on policy citations compared to news mentions, suggesting a potential disconnect between how researchers reference research and how policymakers utilize it. This highlights the need for improved communication strategies to ensure research informs policy decisions more effectively. This study provides valuable insights for researchers, policymakers, and science communicators. Researchers can tailor their dissemination efforts to reach policymakers through media channels. Policymakers can leverage these findings to identify research with higher policy relevance. Science communicators can play a critical role in translating research for policymakers and fostering dialogue between the scientific and policymaking communities.

Motivation & Objective

• To investigate the factors influencing the citation of scientific research in policy documents across a large corpus of policy journals.
• To identify the relative influence of scientific citations, media mentions, and open access status on policy impact.
• To examine whether publication year, policy type, or research discipline significantly affect policy citation likelihood.
• To assess the role of science communication channels such as news media and social media in enhancing research uptake by policymakers.
• To inform strategies for improving the translation of research into policy through targeted dissemination and communication practices.

Proposed method

• Collected data on 124,986 articles from 434 public policy journals, focusing on citations in policy documents.
• Used Altmetric attention scores to measure media mentions, social media engagement, and news coverage.
• Integrated Overton data to assess the presence and type of open access (e.g., hybrid vs. fully open access) for each article.
• Applied multivariate logistic regression to model the probability of a research article being cited in policy documents.
• Controlled for publication year, journal impact factor, and research discipline to isolate the effect of communication and access factors.
• Validated findings using robustness checks across subgroups and sensitivity analyses to ensure reliability of results.

Experimental results

Research questions

• RQ1 Which communication and access factors most strongly predict the likelihood of a research article being cited in policy documents?
• RQ2 How do media mentions and social media engagement compare in influencing policy citations relative to academic citations?
• RQ3 Does the type of open access model (e.g., hybrid vs. fully open access) affect the probability of policy citation?
• RQ4 To what extent do publication year and policy type moderate the relationship between research visibility and policy impact?
• RQ5 How does the influence of news mentions compare to academic citations in predicting policy citations?

Key findings

• News mentions and social media engagement significantly increase the odds of a research article being cited in policy documents, with news mentions showing a stronger effect than academic citations.
• Articles with non-payed access (e.g., subscription-based or hybrid open access) have a higher likelihood of policy citation compared to fully open access articles, suggesting access model matters.
• Paywalled articles show lower policy citation likelihood, indicating that restricted access reduces policy relevance despite academic visibility.
• Academic citations have a weaker predictive power for policy citations than media mentions, revealing a potential disconnect between scholarly and policy communities.
• Publication year and policy type show no statistically significant influence on the likelihood of policy citation, suggesting temporal and sectoral neutrality in policy uptake.
• The combination of media visibility and open access (excluding fully open access) yields the highest probability of policy citation, underscoring the importance of strategic dissemination.