From b00728db81f888713030080e2061f0e45056e715 Mon Sep 17 00:00:00 2001 From: robertd17 <32734387+robertd17@users.noreply.github.com> Date: Thu, 22 Aug 2024 15:16:09 +0100 Subject: [PATCH] Update guide-intro.qmd --- guide-intro.qmd | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/guide-intro.qmd b/guide-intro.qmd index 4a7fc4c..d79c792 100644 --- a/guide-intro.qmd +++ b/guide-intro.qmd @@ -2,7 +2,7 @@ ## Introduction -This introduction explains the rationale for recognising and rewarding open research in the assessment of researchers, with reference to the open research and responsible research assessment agendas that have evolved in recent years. It can be used as a reference text for a group of stakeholders undertaking a self-assessment exercise using the OR4 [maturity framework](maturity-framework.qmd), and to inform engagement in support of planned action, such as the development of business cases, dialogue with key stakeholders, and communications with the wider research community. It can serve to establish a shared understanding of open research and responsible research assessment, and an awareness of the drivers for strategic action in these areas in the higher education and research sector. +This introduction explains the rationale for recognising and rewarding open research in the assessment of researchers, with reference to the open research and responsible research assessment agendas that have evolved in recent years. It can be used as a reference text for a group of stakeholders undertaking a self-assessment exercise using the OR4 [maturity framework](maturity-framework.qmd), and to inform engagement in support of planned action, such as the development of business cases, consultation with key stakeholders, and communications with the wider research community. It can serve to establish a shared understanding of open research and responsible research assessment, and an awareness of the drivers for strategic action in these areas in the higher education and research sector. ## Summary @@ -16,10 +16,10 @@ This broader agenda is shaping the national research assessment framework in the Institutions therefore need to start developing researcher assessment policies to integrate recognition and reward for open research. This work will involve a number of challenges of implementation, which can be characterised at four levels: -- political: getting buy-in from institutional leaders and managers and key stakeholders in professional services to support policy adoption and implementation across relevant procedures -- cultural: securing assent from members of the research community in their capacity as both assessors and subjects of research assessment to the inclusion of open research in research assessment and bringing about changes in practice; -- practical: defining open research in such a way that instances of it can be demonstrated, identified, validated, and qualitatively evaluated within the context of an overall assessment, and providing the training and guidance that enables researchers and assessors to use the criteria effectively; -- operational: implementing the changes to policies and procedures and underpinning systems, processes and support, creating and delivering guidance and training, and monitoring and managing compliance with implemented policies. +- **political**: getting buy-in from institutional leaders and managers and key stakeholders in professional services to support policy adoption and implementation across relevant procedures +- **cultural**: securing assent from members of the research community in their capacity as both assessors and subjects of research assessment to the inclusion of open research in research assessment and bringing about changes in practice; +- **practical**: defining open research in such a way that instances of it can be demonstrated, identified, validated, and qualitatively evaluated within the context of an overall assessment, and providing the training and guidance that enables researchers and assessors to use the criteria effectively; +- **operational**: implementing the changes to policies and procedures and underpinning systems, processes and support, creating and delivering guidance and training, and monitoring and managing compliance with implemented policies. The OR4 implementation guide addresses these aspects of implementation, with an emphasis on the political and cultural aspects in the earlier sections moving into the practical and operational aspects in the later sections. @@ -37,7 +37,7 @@ UKRI [describes open research](https://www.ukri.org/what-we-do/supporting-health Open research can be situated in the context of a global discourse about open knowledge and openness in academic practice that often uses the term open science. In the open knowledge paradigm, 'openness' is integral to the practices by which research is conducted, communicated, evaluated, validated and instrumentalised. Open research practice is held to have a direct relationship to research integrity (through transparency of methods and outputs), research quality (through the use of evidentiary and reproducible practices), sustainability (through use of appropriate standards and formats, preservation infrastructure, and persistent identifiers), and reach and impact (through the accessibility and re-usability of outputs). ::: -## Why shuold open research be part of research assessment? +## Why should open research be part of research assessment? There are key reasons why open research practices are important in the context of research assessment: @@ -47,27 +47,29 @@ There are key reasons why open research practices are important in the context o - The broader range of activities and outputs associated with open research practices facilitates a more rounded assessment of a researcher's activities and outputs than is possible where publications are the primary or exclusive focus of assessment. -A researcher who uses open research practices better demonstrates and enables verification of the quality of their research, maximises the potential of their research to generate value, and is able to provide a more representative picture of their research activities. +A researcher who uses open research practices better demonstrates and enables verification of the quality of their research, maximises the potential of their research to generate value, and is able to provide a more representative picture of the totality of their research and related activities. -Operationalisation of open research incentives and expectations will signal that open practices are considered to be an essential part of how research is carried out. It will power the adoption of open research practices by researchers and lead to improvements in research integrity, quality and impact. +Operationalisation of open research incentives and expectations will signal that open practices are considered by an institution to be an essential part of how research is carried out. It will power the adoption of open research practices by researchers and lead to improvements in research integrity, quality and impact. ## Open research principles are widely accepted but not fully integrated into research practice -The importance of openness in research is acknowledged by governments, funders, and research-performing institutions. The principles of open research have gained widespread acceptance in recent years. The 2021 adoption by the UNESCO member states of its [Recommendation on Open Science](https://doi.org/10.54677/MNMH8546) marks a significant milestone in this respect. Many public research funders and most research institutions in the UK have established policies on open access to research publications and the management and sharing of research data, which are fundamental open research practices. More recently, some institutions have adopted [statements in support of open research](https://www.ukcorr.org/2020/12/02/open-access-is-not-enough-reproducible-science-research-and-scholarship/), endorsing the principles and aims of open research and encouraging use of relevant open research practices. +The principles of open research have gained widespread acceptance in recent years, and the importance of openness in research is acknowledged by governments, funders, and research-performing institutions. The 2021 adoption by the UNESCO member states of its [Recommendation on Open Science](https://doi.org/10.54677/MNMH8546) marks a significant milestone in this respect. Many public research funders and most research institutions in the UK have established policies on open access to research publications and the management and sharing of research data, which are fundamental open research practices. More recently, some institutions have adopted [statements in support of open research](https://www.ukcorr.org/2020/12/02/open-access-is-not-enough-reproducible-science-research-and-scholarship/), endorsing the principles and aims of open research and encouraging use of relevant open research practices. But open research policies and statements are as yet relatively unintegrated into institutional research strategy and planning and actual research practice. Beyond high levels of compliance with open access mandates, driven in large part by the requirements of the UK's Research Excellence Framework (REF), there is little evidence of widespread open research practice. Rates of effective data sharing are still low.^[See e.g.: Gabelica, M., Bojčić, R. and Puljak, L. (2022), 'Many researchers were not compliant with their published data sharing statement: a mixed-methods study'. Journal of Clinical Epidemiology, 150: 33-41. ; Lucas-Dominguez, R. et al (2021), 'The sharing of research data facing the COVID-19 pandemic'. Scientometrics (2021). .] Open research practices are for the most part not incentivised and rewarded; nor, with the exception of open access publication, are they systematically monitored or enforced, either by institutions or by the funders of research. ## Systems of reward and recognition can drive changes in researcher behaviour and academic cultures -At present, very few institutional recruitment, promotion, probation and appraisal frameworks include reference to open research criteria or outputs other than research publications; standards and practices for evidencing a track record in open research are not well-established; and there is a lack of guidance, training and support related to open research for researchers and staff involved in assessment. In consequence, use of open research practices is rarely evidenced or considered in the formal assessment activities, and is not monitored by institutions.^[See: Pontika, N. et al. (2021), 'ON-MERRIT D6.1 Investigating institutional structures of reward & recognition in Open Science & RRI (1.0)'. Zenodo. ; Khan, H. et al. (2022), 'Open science failed to penetrate academic hiring practices: a cross-sectional study'. Journal of Clinical Epidemiology, 144: 136-143. .] +At present, very few institutional recruitment, promotion, probation and appraisal frameworks include reference to open research criteria or outputs other than research publications; standards and practices for evidencing a track record in open research are not well-established; and there is a lack of guidance, training and support related to open research for researchers and staff involved in assessment. In consequence, use of open research practices is rarely evidenced or considered in the formal assessment activities, and is in large part unmonitored by institutions.^[See: Pontika, N. et al. (2021), 'ON-MERRIT D6.1 Investigating institutional structures of reward & recognition in Open Science & RRI (1.0)'. Zenodo. ; Khan, H. et al. (2022), 'Open science failed to penetrate academic hiring practices: a cross-sectional study'. Journal of Clinical Epidemiology, 144: 136-143. .] -With momentum for research assessment reform building globally, there is an opportunity to integrate open research into revised researcher assessment frameworks and practices. Universities play a critical role in the system of academic reward and recognition. It is in their power to include open research criteria in their recruitment specifications, probation objectives and promotion frameworks, performance and development review processes, and research planning activities. By this means researchers can be incentivised and supported to build a track record in open research and to present that track record in an assessment activity, while assessment practices can recognise and give credit for a record of open research practice. This will drive increased adoption of open research practices, and, in time, bear fruit in the recruitment and promotion of staff who are recognised for working in ways that increase the integrity, quality and impact of the institution's research output. +With momentum for research assessment reform building globally, there is an opportunity to integrate open research into revised researcher assessment frameworks and practices. Universities play a critical role in the systems of academic reward and recognition. It is in their power to include open research criteria in their recruitment specifications, probation objectives and promotion frameworks, performance and development review processes, and research planning activities. By this means researchers can be incentivised and supported to build a track record in open research and to present that track record in an assessment activity, while assessment practices can recognise and give credit for a record of open research practice. This will drive increased adoption of open research practices, and, in time, bear fruit in the recruitment and promotion of staff who are recognised for working in ways that increase the integrity, quality and impact of the institution's research output. -## The history of research assessment reform can be characterised in terms of an evolution from an agenda focused almost exclusively on the use of publication-based metrics towards a broader framework of responsible research assessment (Figure 1). This broader, more instrumental agenda considers research assessment as a means of enabling the best researchers to flourish, promoting diversity and inclusion, and supporting the production of high-quality research – in short, as a means to engineer research culture. Within this agenda, there has been growing attention to the role of open research practices in relation to research assessment. +## Open research and research assessment reform + +The history of research assessment reform can be characterised in terms of an evolution from an agenda focused almost exclusively on the use of publication-based metrics towards a broader framework of responsible research assessment (Figure 1). This broader, more instrumental agenda considers research assessment as a means of enabling the best researchers to flourish, promoting diversity and inclusion, and supporting the production of high-quality research – in short, as a means to engineer research culture. Within this agenda, there has been growing attention to the role of open research practices in relation to research assessment. ![Figure 1. Milestones in the history of research assessment reform](images/figures/researchassessment-timeline.jpg) -The [San Francisco Declaration on Research Assessment](https://sfdora.org/read/) (DORA, 2013), the founding text of research assessment reform, was primarily concerned with research publications and related metrics, although its second recommendation for institutions states: +Although the [San Francisco Declaration on Research Assessment](https://sfdora.org/read/) (DORA, 2013), the founding text of research assessment reform, was primarily concerned with research publications and related metrics, its second recommendation adumbrates a broader assessment agenda: > For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice. @@ -85,7 +87,7 @@ In the [Agreement on Reforming Research Assessment](https://coara.eu/agreement/t ## National and institutional assessment practices need to develop -The greater emphasis on open research in the research assessment reform agenda is relatively recent, and national and institutional research assessment policies have so far reflected a prevailing focus on publications and the responsible use of publication metrics. In a survey undertaken by the OR4 project in 2023, 44 or 73% of 70 institutions stated that they had a responsible research assessment statement or policy. The majority of these were focused on the responsible use of publication metrics.^[Barnett, J. at al. (2024). 'OR4 Research Assessment Survey Report'. Working Paper No 5. .] In scope and terminology many of these statements follow and reference DORA and the Leiden Manifesto. +The greater emphasis on open research in the research assessment reform agenda is relatively recent, and national and institutional research assessment policies have so far reflected a prevailing focus on publications and the responsible use of publication metrics. In a survey undertaken by the OR4 project in 2023, 44 or 73% of 60 institutions stated that they had a responsible research assessment statement or policy. The majority of these were focused on the responsible use of publication metrics.^[Barnett, J. at al. (2024). 'OR4 Research Assessment Survey Report'. Working Paper No 5. .] In scope and terminology many of these statements follow and reference DORA and the Leiden Manifesto. The almost exclusive focus on the assessment of research publications is understandable, given their prominence in the systems of academic recognition and reward. In REF 2021, of 185,353 outputs submitted, 180,509 or 97.4% fell into the main academic publications categories A-E (including authored and edited books, book chapters, journal articles and conference contributions). 154,826 outputs or 83.5% of the total were journal articles. The number of research data sets and databases submitted was 31; the number of software outputs was 11 (Figure 2).^[REF 2021 Submitted outputs' details. .] @@ -97,7 +99,7 @@ But the landscape is beginning to change, and institutions will need to develop ## The changing national and international research assessment environment -In comparison to previous national assessment exercises, REF 2029 places greater emphasis on institutional research culture, including use of open research practices. Institutions can provide evidence of this in the 'People, Culture and Environment' element, while the element 'Contribution to Knowledge and Understanding' enables a greater diversity of research activities and outputs to be evidenced.^[Research England (2023), 'Research Excellence Framework 2028: Initial decisions and issues for further consultation'. .] +In comparison to previous national assessment exercises, REF 2029 places greater emphasis on institutional research culture, including use of open research practices. Institutions can provide evidence of this in the People, Culture and Environment element, while the element Contribution to Knowledge and Understanding enables a greater diversity of research activities and outputs to be evidenced.^[Research England (2023), 'Research Excellence Framework 2028: Initial decisions and issues for further consultation'. .] As the national research assessment framework progressively assimilates open research objectives and assessment criteria, institutions will be obliged to conform, and to integrate open research into their own policies, systems and processes. The greater emphasis on open research in the 2029 REF is consistent with international developments, where other national research environments are also beginning to demonstrate greater alignment to the principles embodied in the Agreement on Reforming Research Assessment, and to identify open research as an important element of incentive and reward systems.