Multidimensional impact of research: developing and testing a model for assessment
Main Article Content
Abstract
In recent decades, changes in science have been characterized by internationalization and the quest for impact. This article introduces the impact assessment process of 23 research projects of the Capes prInt Program aimed at internationalizing Brazilian science. This work developed an impact assessment model, applying it experimentally. The development was based on scientific texts on modes of production of scientific knowledge and impact assessment. The assessment revealed that, despite the COVID-19 pandemic, the projects have advanced in their internationalization goals. As for the impacts, those related to science and education predominated, with a significant number of projects also indicating impacts on public policies and organizational practices. The article adds to understanding modes of knowledge production and research impact measurement. Furthermore, it can be useful to directors of research institutions, research managers, and researchers interested in measuring the research impact on organizations and society.
Downloads
Metrics
Article Details
![Creative Commons License](http://i.creativecommons.org/l/by/4.0/88x31.png)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Cadernos EBAPE.BR undertakes to contribute to the protection of authors’ intellectual rights. On this matter:
- It uses the Creative Commons BY (CC-BY) license for all texts it publishes, except when there is indication of specific holders of copyrights and property rights;
- It uses the similarity verification software of content - Plagiarism (Crossref Similarity Check);
- It takes actions to fight against plagiarism and ethical misconduct aligned with the guidelines of the Committee on Publication Ethics (COPE).
Further information on the Code of Ethics adopted by Cadernos EBAPE.BR can be found in Ethical Standards and Code of Conduct.
References
Aguinis, H., Shapiro, D. L., Antonacopoulou, E. P., & Cummings, T. G. (2014). Scholarly impact: a pluralist conceptualization. Academy of Management Learning and Education, 13(4), 623-639. Recuperado de http://dx.doi.org/10.5465/amle.2014.0121
Amabile, T. M., Patterson, C., Mueller, J., Wojcik, T., Odomirok, P., Marsh, M. … Kramer, S. (2001). Academic-practitioner collaboration in management research: a case of cross-profession collaboration. Academy of Management Journal, 44(2), 418-431. Recuperado de http://doi.org/10.5465/3069464
Anderson, N. (2007). The practitioner-researcher divide revisited: strategic-level bridges and the roles of IWO psychologists. Journal of Occupational and Organizational Psychology, 80(2), 175-183. Recuperado de http://doi.org/10.1348/096317907X187237
Astley W. G., & Zammuto, R. F. (1992). Organization science, managers, and language games. Organization Science, 3(4), 443-460. Recuperado de https://doi.org/10.1287/orsc.3.4.443
Baldridge, D. C., Floyd, S. W., & Markóczy, L. (2004). Are managers from Mars and academicians from Venus? Toward an understanding of the relationship between academic quality and practical relevance. Strategic Management Journal, 25(11), 1063-1074. Recuperado de http://doi.org/10.1002/smj.406
Barbosa, J. C., Ferreira, M. F., Paiva, C. C., Patrício, K. P., Silva, D. H. S., & Portela, J. C. (2022). A experiência da UNESP com a Agenda 2030: a governança universitária como indutora de ações e articulações para o enfrentamento dos desafios locais e globais. Revista Latinoamericana de Ciencias de la Comunicación, 21(41), 132-146. Recuperado de http://revista.pubalaic.org/index.php/alaic/article/view/945
Bello, D. C., & Kostova, T. (2012). From the Editors: Conducting high impact international business research: The role of theory. Journal of International Business Studies, 43(6), 537-543. Recuperado de https://10.1057/jibs.2012.14
Bloom, N., & Van Reenen, J. (2006). Measuring and explaining management practices across firms and countries (Discussion Paper n. 716). London, UK: London School of Economics and Political Science.
Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on 8 International Journal of Qualitative Methods data from the UK Research Excellence Framework (REF). Journal of Informetrics, 13(1), 325-340. Recuperado de https://doi.org/10.1016/j.joi.2019.01.008
Costa, F. J., Machado, M. A. V., & Câmara, S. F. (2022). Por uma orientação ao impacto societal da pós-graduação em administração no Brasil. Cadernos EBAPE.BR, 20(6), 823-835. Recuperado de https://doi.org/10.1590/1679-395120210222
Eisenhardt, K., Graebner, M. E., & Sonenstein, S. (2016). From the editors: grand challenges and inductive methods: rigor without rigor mortis. Academy of Management Journal, 59(4), 1113-1123. Recuperado de https://doi.org/10.5465/amj.2016.4004
Etzkowitz, H., & Leydesdorff, L. (1995). The triple helix university-industry-government relations: a laboratory for knowledge based economic development. EASST Review, 14(1), 14-19. Recuperado de https://ssrn.com/abstract=2480085
Etzkowitz, H., Webster, A., Gebhardt, C., & Terra, B. R. (2000). The future of the university and the university of the future: evolution of ivory tower to entrepreneurial paradigm. Research Policy, 29(2), 313-330. Recuperado de https://doi.org/10.1016/S0048-7333(99)00069-4
Funtowicz, S. O., & Ravetz, J. R. (1993). Science for the post-normal age. Futures, 25(7), 739-755. Recuperado de https://doi.org/10.1016/0016-3287(93)90022-L
George, G. (2016). From the editor: management research in AMJ: celebrating impact while striving for more. Academy of Management Journal, 59(6), 1880-1895. Recuperado de https://doi.org/10.5465/amj.2016.4006
George, G., Howard-Grenville, J., Aparna, J., & Tihanyi, L. (2016). Understanding and tackling societal grand challenges through management research. Academy of Management Journal, 59(6), 1869-1877. Recuperado de https://doi.org/10.5465/amj.2016.4007
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies. London, UK: Sage Publishing House.
Haley, U. C. V., Page, M.C., Pitsis, T.S., Rivas, J. L., & Yu. K. F. (2017). Measuring and achieving scholarly impact: a report from the Academy of Mangement’s Practice Theme Committee. Recuperado de https://doi.org/10.13140/RG.2.2.11747.86561
Hodgkinson, G. P. (2006). The role of JOOP (and other scientific journals) in bridging the practitioner-researcher divide in industrial, work and organizational (IWO) psychology. Journal of Occupational and Organizational Psychology, 79(2), 173-178. Recuperado de https://doi.org/10.1348/096317906X104013.
Hodgkinson, G. P., Herriot, P., & Anderson, N. (2001). Re-aligning the stakeholders in management research: lessons from industrial, work and organizational psychology. British journal of Management, 12(s1), S41-S48. Recuperado de https://doi.org/10.1111/1467-8551.12.s1.5
Hodgkinson, G. P., & Starkey, K. (2011). Not simply returning to the same answer over and over again: reframing relevance. British Journal of Management, 22(3), 355-369. Recuperado de https://doi.org/10.1111/j.1467-8551.2011.00757.x
Huff, A. (2000). 1999 presidential address: changes in organizational knowledge production. Academy of Management Review, 25(2), 288-293. Recuperado de https://doi.org/10.5465/amr.2000.3312916
Kieser A., & Leiner L. (2009). Why the rigor-relevance gap in management research is unbridgeable. Journal of Management Studies, 46(3), 516-533. Recuperado de https://doiorg/101111/j1467-6486200900831x
Kreiling, L., & Paunov, C. (2021). Knowledge co-creation in the 21st century: a cross-country experience-based policy report (OECD Science, Technology and Industry Policy Papers, n. 115). Paris, France: OECD Publishing. Recuperado de https://www.oecd.org/innovation/knowledge-co-creation-in-the-21st-century-c067606f-en.htm
Lauronen, J. P. (2020). The dilemmas and uncertainties in assessing the societal impact of research. Science and Public Policy, 47(2), 207-218. Recuperado de https://doi.org/10.1093/scipol/scz059
Lazzarini, S. (2017). Pesquisa em Administração: em busca de impacto social e outros impactos. Revista de Administração de Empresas, 57(6), 620-625. Recuperado de https://doi.org/10.1590/S0034-759020170608
Manville, C., Jones, M. M., Henham, M. L., Castle-Clarke, S., Frearson, M., Gunashekar, S. … Grant, J. (2015). Preparing impact submissions for REF 2014: an evaluation approach and evidence. Santa Monica, CA: Rand Europe. Recuperado de https://www.rand.org/pubs/research_reports/RR726.html
Mendes-da-Silva, W. (2019). Convergência, comunicação, e impacto da pesquisa em negócios. Revista de Administração Contemporânea, 23(1), 1-7. Recuperado de https://doi.org/10.1590/1982-7849rac2019180346
Ministério da Educação e Cultura. (2019, outubro 24). Novo modelo de avaliação medirá impacto social e inserção regional das pesquisas. Recuperado de http://portal.mec.gov.br/setec-programas-e-acoes/acordo-gratuidade/225-noticias/sistemas-1375504326/81611-novo-modelo-de-avaliacao-medira-impacto-social-e-insercao-regional-das-pesquisas
Muhonen, R., Benneworth, P., & Olmos-Penuela, J. (2020). From productive interactions to impact pathways: understanding the key dimensions in developing SSH research societal impact. Research Evaluation, 29(1), 34-47. Recuperado de https://doi.org/10.1093/reseval/rvz003
Pettigrew, A. (2001). Management research after modernism. British Journal of Management, 12(s1), s61-s70. Recuperado de https://doi.org/10.1111/1467-8551.12.s1.8
Pettigrew, A. (2011). Scholarship with impact. British Journal of Management, 22(3), 347-354l. Recuperado de https://doi.org/10.1111/j.1467-8551.2011.00769.x
Razmgir, M., Panahi, S., Ghalichi, L., Mousavi, S. A. J., & Sedghi, S. (2021). Exploring research impact models: a systematic scoping review. Research Evaluation, 30(4), 443-457. Recuperado de https://doi.org/10.1093/reseval/rvab009
Reale, E., Avramov, D., Canhial, K., Donovan, C., Flecha, R., Holm … Van Horik, R. (2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298-308. Recuperado de https://doi.org/10.1093/reseval/rvx025
Reed, M. S., Ferre, M., Martin-Ortega, J., Blanche, R., Lawford-Rolfe, R., Dallimer, M. … Holden, J. (2021). Evaluating impact from research: A methodological framework. Research Policy, 50(4), 104147. Recuperado de https://doi.org/10.1016/j.respol.2020.104147
Research Excellence Framework. (2020, outubro). Index of revisions to the ‘panel criteria and working methods’ (2019/02). Recuperado de https://www.ref.ac.uk/media/1450/ref-2019_02-panel-criteria-and-working-methods.pdf
Rolfsen, M., Johnsen, A., & Knutstad, G. (2007). Action engagement: improving researchers’ involvement in action research projects. Systemic Practice and Action Research, 20(1), 53-63. Recuperado de https://doi.org/10.1007/s11213-006-9049-x
Rynes, S. L. (2007). Let’s create a tipping point: what academics and practitioners can do, alone and together. Academy of Management Journal, 50(5), 987-1008. Recuperado de https://doi.org/10.5465/AMJ.2007.27156169
Rynes, S. L., Bartunek, J. M., & Daft, R. L. (2001). Across the great divide: knowledge creation and transfer between practitioners and academics. Academy of management Journal, 44(2), 340-355. Recuperado de https://doi.org/10.5465/3069460
Samuel, G. N., & Derrick, G. E. (2015). Societal impact evaluation: exploring evaluator perceptions of the characterization of impact under the REF2014. Research Evaluation, 24(3), 229-241. Recuperado de https://doi.org/10.1093/reseval/rvv007
Sandes-Guimarães, L. V. D., & Hourneaux, F., Jr. (2020). Research impact – what is it, after all? RAUSP Management Journal, 55(3), 283-287. Recuperado de https://doi.org/10.1108/RAUSP-07-2020-202
Smith, R. (2001). Measuring the social impact of research: difficult but necessary. BMJ, 323(7312), 528. Recuperado de https://doi.org/10.1136/bmj.323.7312.528
Starkey, K. & Madan, P. (2001), Bridging the relevance gap: aligning stakeholders in the future of management research. British Journal of Management, 12(1), s3-s26. Recuperado de https://doi.org/10.1111/1467-8551.12.s1.2
Tenkasi R. V., & Hay, G. W. (2004). Actionable knowledge and scholar-practitioners: a process model of theory-practice linkages. Systemic Practice and Action Research, 17(3), 177-206. Recuperado de https://doi.org/10.1023/B:SPAA.0000031697.76777.ac
The Association to Advance Collegiate Schools of Business. (2012). Impact of research: a guide for business schools. Tampa, FL: AACSB International. Recuperado de https://www.aacsb.edu/insights/reports/impact-of-research-a-guide-for-business-schools
Ventura, A. C., Davel, E. P. B. (2021). Impacto socioambiental da pesquisa. Organizações & Sociedade, 28(99), 710-721. Recuperado de https://doi.org/10.1590/1984-92302021v28n9900PT
Vogel, I. (2012). Review of the use of ‘theory of change’ in international development. London, UK: Department for International Development. Recuperado de https://gsdrc.org/document-library/review-of-the-use-of-theory-of-change-in-international-development/
Warwick Institute for Global Sustainable Development. (2021). URBE Latam theory of change at project proposal stage (UKRI GCRF). London, UK: Warwick University. Recuperado de https://warwick.ac.uk/fac/arts/schoolforcross-facultystudies/igsd/resources/hp-contents/urbe_latam_theory_of_change_-_proposal_stage.pdf
Wickert, C., Post, C., Doh, J. P., Prescott, J. E., & Prencipe, A. (2021). Management research that makes a difference: broadening the meaning of impact. Journal of Management Studies, 58(2), 297-320. Recuperado de https://doi.org/10.1111/joms.12666
Willmott, H. (2012). Reframing relevance as “social usefulness”: a comment on Hodgkinson and Starkey’s “Not simply returning to the same answer over and over again”. British Journal of Management, 23(4), 598-604. Recuperado de https://doi.org/10.1111/j.1467-8551.2012.00839.x
Wood, T., Jr., & Caldas, M. P. (2020). Posfácio: o desafio da transformação na UNESP. In S. R. Valentini, & S. R. Nobre (Orgs.), Universidade em transformação: lições das crises (pp. 445-453). São Paulo, SP: Editora UNESP.
Wood, T., Jr., Holz, E. B., & Souza, R. (2022). When rigor meets relevance: the development of hybrid actionable knowledge production systems. Systemic Practice and Management Research, 36, 1-29. Recuperado de https://doi.org/10.1007/s11213-022-09596-x
Wood, T., Jr., Souza, R., & Caldas, M. O. (2022), The relevance of management research debate: a historical view – 1876-2018. Journal of Management History, 28(3), 409-427. Recuperado de https://doi.org/10.1007/s11213-022-09596-x
Ziman, J. (1996). ‘Postacademic science’: constructing knowledge with networks and norms. Science Studies, 9(1), 67-80. Recuperado de https://doi.org/10.23987/sts.55095