Evaluating the Policy Impact of COVID-19 Health Researches Using the Altmetrics Approach: Countries Policymaking Experiences

Document Type : Research Paper

Authors

1 Assistant Professor, Department of Scientometrics, National Research Institute for Science Policy (NRISP), Tehran, Iran.

2 PhD. in Information Science and Epistemology, Faculty of Literature and Humanities, Science and Research Branch, Islamic Azad University, Tehran, Iran

3 PhD. Student, Department of Medical Library and Information Sciences, School of Health Management & Information Sciences, Iran University of Medical Sciences, Tehran, Iran.

Abstract

Objective: The main objective of this paper is to evaluate the policy impact of health researches in the field of COVID-19 with an Altmetrics approach, and other objectives of the study include the study of the popularity of this subject in social media, governance systems and policy documents that have benefited from these studies, their thematic dimensions, and the policy impact of COVID-19 clinical trial researches.
Method: This was applied research conducted using altmetric, and content analysis indicators. Based on the above, all the researches of COVID-19 in Pubmed, Scopus, Web of Science, and Cochrane was extracted and observed in the Altmetrics database which returned 4308 researches in COVID-19.
Results: 95% of COVID-19 research was mentioned on Twitter. Among them, %0.01 referred to policy documents, of which %45 were related to Switzerland and %36.6 to the United States. Seven of these were conducted as a clinical trial. Topics covered in policy documents of COVID-19 included public health services, clinical sciences, and medical microbiology.
Conclusion: Using altmetric and web metrics are useful in tracking the cultural, social, and policy dimensions of researches and monitoring user interactions on social media, especially Twitter, which is an attractive media for academic and non-academic users in a wide range of medical sciences. Moreover, the mentions of these researches in policy documents, in such a short time, express the importance of them at the high policymaking levels of countries, specifically those in clinical trials.

Keywords

Main Subjects


قلی‌پور، ر.، پورسید، ب.، حمیدی‌زاده، ع.، و امیری، ع. (1389). بررسی تأثیر سیاست پژوهی در فضای سیاست‎گذاری (مطالعه‌ی موردی در مرکز پژوهش‌‌های مجلس شورای اسلامی). مدیریت دولتی، 2(4)، 144-127.
کشاورزی، س.، خواجه‌نوری، ب.، و غفاری‌نسب، ا. (1396). ارزیابی تأثیر رسانه‌‌های مختلف بر میزان مشارکت در جنبش‌‌های محیط‌زیستی: مورد جنبش رفتگران طبیعت ایران. مطالعات راهبردی سیاست‌گذاری عمومی، 7(25)، 92-73.
نامداریان، ل. (1395). بررسی و تبیین کاربردپذیری نتایج پژوهشی در سیاست‌گذاری: پلی می‌ان نظر و عمل. سیاست‌گذاری عمومی، 2(3)، 117-101.
References
Altmetric. (2020a). Text mining: How does Altmetric text mining work? Retrieved fromhttps://help.altmetric.com/support/solutions/articles/6000240263-text-mining
Altmetric. (2020b). Tracking output. Required metadata for content tracking.Retrieved fromhttps://help.altmetric.com/support/solutions/articles/6000240582-required-metadata-for-content-tracking
Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123-44. [DOI:10.1007/s11192-015-1565-y]
Bornmann, L., & Haunschild, R. (2017). Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Scientometrics, 110(2), 937-43. [DOI:10.1007/s11192-016-2200-2] [PMID] [PMCID]
Bornmann, L., & Haunschild, R. (2019). Societal impact measurement of research papers. In W. Glänzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators. Springer handbooks (pp. 609-632). Cham: Springer. [DOI:10.1007/978-3-030-02511-3_23]
Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477-95. [DOI:10.1007/s11192-016-2115-y] [PMID] [PMCID]
Bornmann, L., Haunschild, R., & Patel, V. M. (2020). Are papers addressing certain diseases perceived where these diseases are prevalent? The proposal to use Twitter data as social-spatial sensors. arXiv, arXiv:2004.13974.https://arxiv.org/ftp/arxiv/papers/2004/2004.13974.pdf
Califf, R. M., Zarin, D. A., Kramer, J. M., Sherman, R. E., Aberle, L. H., & Tasneem, A. (2012). Characteristics of clinical trials registered in clinicaltrials.gov, 2007-2010. JAMA, 307(17), 1838-47. [DOI:10.1001/jama.2012.3424] [PMID]
Chow, S. C., & Liu, J. P. (2008). Design and analysis of clinical trials: Concepts and methodologies. Hoboken, NJ: John Wiley & Sons. https://books.google.com/books?id=TQ6g4ADtRLgC&dq
Cronin, B., & Sugimoto, C. R., (Eds.). (2014). Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact. Cambridge, MA: MIT Press. https://books.google.com/books?id=7BSaAwAAQBAJ&dq
Dukeshire, S., & Thurlow, J. (2002). Understanding the link between research and policy. Retrieved from https://docplayer.net/142310-Understanding-the-link-between-research-and-policy.html
Fang, Z., & Costas, R. (2020). Studying the accumulation velocity of altmetric data tracked by Altmetric.com. Scientometrics, 123, 1077-101. [DOI:10.1007/s11192-020-03405-9] [PMID]
Forrester, J. A., Forrester, J. D., & Wren, S. M. (2018). Trends in country-specific surgical randomized clinical trial publications. JAMA Surgery, 153(4), 386-8. [DOI:10.1001/jamasurg.2017.4867] [PMID] [PMCID]
Friedman, L. M., Furberg, C., DeMets, D. L., Reboussin, D. M., & Granger, C. B. (2015). Fundamentals of clinical trials. Cham: Springer. [DOI:10.1007/978-3-319-18539-2]
Gholi Pour, R., Pourseyed, B., Hamidizadeh, A., & Amiri, A. (2010). [Studying the effect of policy research in policy making area (Persian)]. Journal of Public Administration, 2(4), 127-44. https://jipa.ut.ac.ir/article_21111.html
Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web Handbook of Science and Altmetric data. Scientometrics, 110(3), 1209-16. [DOI:10.1007/s11192-016-2237-2] [PMID] [PMCID]
Henschke, N., Kuijpers, T., Rubinstein, S. M., van Middelkoop, M., Ostelo, R., Verhagen, A., & et al. (2012).Trends over time in the size and quality of randomised controlled trials of interventions for chronic low-back pain. European Spine Journal, 21(3), 375-81. [DOI:10.1007/s00586-011-2023-z] [PMID] [PMCID]
IJsselmuiden, C., & Matlin, S. (2006). Why health research? Retrieved from http://www.cohred.org/downloads/cohred_publications/Why_Health_Research_Research-vol.1_0_0.pdf
Keshavarzi, S., Khaje Noori, B., & Ghaffari Nasab, E. (2018). [Assessment of media influence on the intensity of participation in the environmental movements: The case of nature cleaners movement (Persian)]. Strategic Studies of Public Policy, 7(25), 73-92. http://sspp.iranjournals.ir/article_29727.html
Kousha, K., & Thelwall, M. (2020). COVID-19 publications: Database coverage, citations, readers, tweets, news, Facebook walls, Reddit posts. arXiv, arXiv:2004.10400. https://arxiv.org/abs/2004.10400
Kovic, I., Lulic, I., & Brumini, G. (2008). Examining the medical blogosphere: An online survey of medical bloggers. Journal of Medical Internet Research, 10(3), e28. [DOI:10.2196/jmir.1118] [PMID] [PMCID]
Lacey, J., Howden, M., Cvitanovic, C., & Colvin, R. M. (2018). Understanding and managing trust at the climate science-policy interface. Nature Climate Change, 8, 22-8. [DOI:10.1038/s41558-017-0010-z]
Lasswell, H. D. (1971). A pre-view of policy sciences. New York: American Elsevier Publishing Company. https://books.google.com/books?id=MY1RAAAAMAAJ&dq
Marston, G., & Watts, R. (2003). Tampering with the evidence: A critical appraisal of evidence-based policy-making. The Drawing Board: An Australian Review of Public Affairs, 3(3), 143-63. https://uq.rl.talis.com/items/994F159F-C235-B576-CA14-EB2A91C2A2F5.html
Moradi, Sh., & Dokhani, F. (2020). Using the Quadruple Helix Model for evaluation of health science researches: Case study of D8 countries. Library Hi Tech, March. [DOI:10.1108/LHT-08-2019-0156]
Moses, H., Matheson, D. H. M., Cairns-Smith, S., George, B. P., Palisch, C., & Ray Dorsey, E. (2015). The anatomy of medical research: US and international comparisons. JAMA, 313(2), 174-89. [DOI:10.1001/jama.2014.15939] [PMID]
Namdarian, L. (2016). [Review and explanation the applicability of research in policy making: A bridge between theory and practice (Persian)]. Public Policy, 2(3), 101-17. [DOI:10.22059/PPOLICY.2016.60111]
National Bureau of Economic Research. (2020). About the NBER. Retrieved from https://www.nber.org/info.html
Newman, J., Cherney, A., & Head, B. W. (2016). Do policy makers use academic research? Reexamining the “two communities” theory of research utilization. Public Administration Review, 76(1), 24-32. [DOI:10.1111/puar.12464]
Newson, R., King, L., Rychetnik, L., Bauman, A. E., Redman, S., Milat, A. J., & et al. (2015). A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: Perceptions of Australian researchers. BMJ Open, 5(7), e008153. [DOI:10.1136/bmjopen-2015-008153] [PMID] [PMCID]
Parsons, W. (2002). From muddling through to muddling Up-evidence based policy making and the modernisation of British Government. Public Policy and Administration, 17(3), 43-60. [DOI:10.1177/095207670201700304]
Pennycook, G., McPhetres, J., Zhang, Y., Lu, J. G., & Rand, D. G. (2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7), 770-80. [DOI:10.1177/0956797620939054]
Pielke Jr, R. A. (2007). The honest broker: Making sense of science in policy and politics. Cambridge: Cambridge University Press. [DOI:10.1017/CBO9780511818110]
Robinson-Garcia, N., van Leeuwen, T. N., & Rafols, I. (2018). Using altmetrics for contextualised mapping of societal impact: From hits to networks. Science and Public Policy, 45(6), 815-26. [DOI:10.1093/scipol/scy024]
Rothman, K.J., Greenland, S., & Lash, T. L. (1998). Types of epidemiologic studies. In K. J. Rothman, S. Greenland. Modern epidemiology. Philadelphia: Lippincott-Raven. https://books.google.co.uk/books?id=cL3zwIFHB7cC&dq
Schuster, M. A., McGlynn, E. A., & Brook, R. H. (1998). How good is the quality of health care in the United States? The Milbank Quarterly, 76(4), 517-63. [DOI:10.1111/1468-0009.00105] [PMID] [PMCID]
Sean. (2019). The detailed guide to understanding your social analytics. Retrieved from https://www.lyfemarketing.com/blog/social-analytics/
Selby, P., & Autier, P. (2011). The impact of the process of clinical research on health service outcomes. Annals of Oncology, 22(Suppl 7), VII5-VII9. [DOI:10.1093/annonc/mdr419] [PMID]
Sullivan, T. (2018). The importance of clinical trials. Retrieved from https://www.policymed.com/2010/05/the-importance-of-clinical-trials.html
Tahamtan, I., & Bornmann, L. (2020). Altmetrics and societal impact measurements: Match or mismatch? A literature review. El Profesional de la Información, 29(1), e290102. [DOI:10.3145/epi.2020.ene.02]
Thelwall, M., & Thelwall, S. (2020). Retweeting for COVID-19: Consensus building, information sharing, dissent, and lockdown life. arXiv, arXiv:2004.02793. https://arxiv.org/abs/2004.02793v2
The World Bank. (2020). About the World Bank. Retrieved from https://www.worldbank.org/en/about
Viergever, R. F., & Li, K. (2015). Trends in global clinical trial registration: An analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open, 5(9), e008932. [DOI:10.1136/bmjopen-2015-008932] [PMID] [PMCID]
Willis, C. D., Riley, B., Stockton, L., Viehbeck, S., Wutzke, S., & Frank, J. (2017). Evaluating theimpact of applied prevention research centres: Results from a modified Delphi approach. Research Evaluation, 26(2), 78-90. [DOI:10.1093/reseval/rvx010]
Wooding, S., Hanney, S., Buxton, M., & Grant, J. (2005). Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology, 44(9), 1145-56. [DOI:10.1093/rheumatology/keh708] [PMID]
Wooding, S., Hanney, S., Pollitt, A., Buxton, M., & Grant, J. (2011). Project retrosight: Understanding the returns from cardiovascular and stroke research: The policy report. Rand Health Quarterly, 1(1), 16. [PMID] [PMCID]
Wouters, P., Zahedi, Z., & Costas, R. (2019). Social media metrics for new research evaluation. In W. Glänzel, H. F. Moed, U. Schmoch, M. Thelwall (Eds.), Springer handbook of science and technology indicators. Springer handbooks (pp. 687-713). Cham: Springer. [DOI:10.1007/978-3-030-02511-3_26]