Research assessment must reform to recognise open and responsible research practices

2022-01-17

by Thomas Klebel, Nancy Pontika, Tony Ross-Hellauer

This content has been originally published on the OpenAIRE blog.

Change is deep underway to make research more transparent and participatory through new open and responsible research practices. Key movements for change include Open Science, such as publishing Open Access (OA) or sharing research data and code, and approaches that make research and innovation more responsive to the needs of society (Responsible Research & Innovation, or RRI). Funders such as the European Commission, with its 95.5 billion Euro funding scheme Horizon Europe, are part of this movement, as increasingly requiring open and responsible practices within the research they fund. Yet, current research assessment processes are increasingly recognised as a barrier to this goal. Potential issues include the overuse of inappropriate indicators, an overemphasis on quantity over quality, and a neglect of open and responsible practices per se.

As part of our Horizon 2020 project ON-MERRIT (investigating issues of equity in open and responsible research), we wanted to quantify the extent to which practices and goals of Open Science and RRI activities are considered by universities across the globe when assessing researchers for promotion and tenure. To this end, we collected and analysed documents on the policies for promotion, review and tenure from 107 academic institutions with a total of 145 policies, across seven countries (Austria, Brazil, Germany, India, Portugal, UK and US). In order to draw meaningful conclusions on the level of countries, we sampled more institutions for countries with larger higher education systems, such as the US and the UK.

What are the most common indicators when assessing researchers?

Across all countries and institutions, by far the most common indicator mentioned in the policies was “service to the profession”, which includes activities such as organising conferences or mentoring PhDs. Other common indicators were ubiquitous concepts such as the “number of publications” or “impact”. On the other end of the spectrum, data sharing was mentioned by no university, and OA publishing by one. Indicators referring to RRI concepts were also quite uncommon, with very few policies mentioning concepts related to gender or citizen science.

Assessment practices are not the same across countries

While institutions still seem to focus on more conventional indicators such as numbers of publications or impact, there are substantial differences between countries. For example, while all Austrian universities mention taking into account the number of publications, this is much less common in the UK . In turn, UK universities also seem not to give much weight to patenting, which is quite common in Brazil. All universities from these two countries, however, mention “service to the profession”, which is much less common in other countries. Finally, only institutions from Austria and Germany consider gender in their policies.

So what?

As it stands, there seems to be much to be done when it comes to including indicators of Open Science and RRI into how institutions reward specific scientific practices. It can be argued (as Elizabeth Gadd recently has) that Open Science principles should not be the target of specific metrics, as they represent basic acts of good research practice. We believe that institutional policies for promotion, review and tenure are an important stepping stone for academia to transition into a world of openly available outputs of academic research. However, it should not be forgotten that various practices (such as data and software sharing) might be better suited to some disciplines than others. Implementing the same policies across all disciplines would only serve to marginalise research that quite simply works differently than some of the more “hard” sciences. Nevertheless, the need for reform could not be clearer, as our findings that open and responsible research practices are rarely considered within promotion and tenure policies clearly show.

More detail on our methods and our results is available in our report Investigating Institutional Structures of Reward & Recognition in Open Science & RRI.

This research has been supported by the project “ON-MERRIT,” funded by the European Commission under the Horizon 2020 program (grant number 824612), which investigates how and if open and responsible research practices could worsen existing inequalities.