Quality thresholds and institutional gaming

UCU has serious concerns about the potential for ‘game playing’ by institutions in submitting their REF returns which may bear little relation to the quality of research of individual members of staff.  We are particularly concerned about the detrimental impact this may have on staff, and the way it may distort the assessment process and future funding decisions.

Institutions will be seeking to maximise the scores received for each Unit of Assessment by the respective panels, and excluding research-active academics staff where they are concerned that this may bring the overall score down.  This partly relates to institutional concerns about league tables that also show the percentage of academic researchers submitted to particular Units of Assessment and factor this into their rankings (as well as overall scores). Indeed, HEIs now seem keener than in previous RAEs to maximise the proportion of relevant staff entered for the REF. Some are doing this by making a distinction between ‘research-active’ and teaching-focused staff and switching staff viewed as not useful to the REF to ‘teaching-focused’ pathways.  

In the REF assessment, outputs will be assessed and graded according to four starred levels. Research regarded as ‘world-leading’ will be rated as 4*; research that is ‘internationally excellent’ will be rated as 3*; research that is ‘internationally recognised’ will be rated as 2*; and research that is ‘nationally recognised’ will be rated as 1*. It appears that a number of institutions have imposed a quality threshold requiring staff to have an average predicted grading of somewhere between 2.5* and 3* for their four submitted outputs. This move relates to announcements by the funding councils that only research rated at 3* and 4* in the REF would be included in the calculations for future funding allocations.

The adoption of quality thresholds based on these gradings means that where an academic’s leading research outputs are regarded in the main as merely ‘internationally recognised’ rather than ‘internationally excellent’ or ‘world leading’, then their institution may regard their work as of insufficient quality to be included in the REF submission. Moreover, researchers who have produced outputs that are regarded as ‘internationally excellent’ or higher, but of an insufficient quantity, may find themselves excluded (one or two excellent books will be worth less than four articles in the ‘right’ journals).

It is important to note that decisions on submissions will be based on predicted REF ratings rather than certainty as to the rating any one output might receive from the REF panel assessment, with the processes adopted by HEIs to make these predictions often not particularly robust.   

The imposition of quality thresholds is part of a wider process of ‘gaming’ on the part of institutions as they seek to maximise outcomes from the REF and the prestige, reputational boost and future funding allocations associated with the REF results.  This could also mean concentrating efforts in a smaller number of research areas, in certain types or areas of research, certain disciplines or sub-fields of these disciplines. In some particular units of assessments, institutions could decide not to make a submission at all, or submit only researchers from a particular research group or only research reflecting a particular research strategy, or to submit a very small elite group of researchers to increase the prospects of appearing higher in REF ‘league tables’. 

Exclusion from the REF submission may relate to the view that an individual’s research is not of sufficient quality, but it may also relate to concerns that an individual’s research does not fit in easily with regards to the Units of Assessment, because it is unorthodox, or because it does not fit into the research profile that the institution or submitting department itself wishes to project. Some research, for example that of an interdisciplinary nature, may be regarded as not fitting neatly within one unit of assessment and also excluded for this reason (this is despite assurance in the funding councils’ guidance that arrangements will be put in place to ensure that interdisciplinary work, and work that does not fit neatly into Unit of Assessment parameters will be assessed on an equal basis). 

Submitting units within institutions are also required to submit a certain ratio of ‘impact’ case studies (demonstrating wider impact of research beyond academia) to researchers submitted to the REF sub-panels. Thus if there are not enough impact case studies suitable for inclusion in the submission, then the institution or submitting unit may decide to reduce the number of researchers in the submission, favouring those who have produced impact case studies over others.

Reports from UCU members and branches indicate that a variety of other dubious practices have been employed by institutions in order to boost their potential REF scores.  This includes the practice of appointing academics with potentially high REF scores on short-term and fractional contracts to cover the REF census date. These might include prominent retired academics or academics with substantive positions outside of the UK. Concerns about this practice have also been reported in the Times Higher. See http://www.timeshighereducation.co.uk/news/cardiff-bakers-dozen-for-expiry-date-check-the-ref/2005132.article

More generally, the REF and the RAE before it have been criticised for creating a transfer market of REF superstars, brought in by institutions to boost their REF scores, and leaving other academics in the particular schools/ departments feeling marginalised.  For a Times Higher report on this practice, and the broader problem of academic staff recruitment being driven by institutional REF strategies, see: http://www.timeshighereducation.co.uk/421623.article

The other side of this coin is that concerns about losing ‘REF-able’ staff to other HEIs has led some institutions to enforce lengthy notice periods which make it difficult for staff to take up new positions once appointed. This has included the practice of institution’s enforcing contractual conditions which mean they can refuse to allow the period between the summer term/second semester ending and the autumn term/first semester beginning as counting towards the calculation of the notice period (meaning notice has to be given for a specific period in advance of the end of the summer term/first semester, and staff giving their notice after this point are then obliged to remain in post until the end of the following semester/term).

These practices undermine the credibility of the REF, and are also illustrative of the detrimental impact the REF has on the HE sector. They are of particular concern given the potential that decisions on REF inclusion have in terms of making and breaking academic careers. As explained elsewhere on the blog, and in UCU guidance, we have called on institutions to issue assurances around no detriment for those staff not included in the REF, given the flawed nature of the process and its unreliability as an indicator of academic ‘performance’. 

We are also seeking to gather together commentaries and case studies on this blog on the way in which HEIs are handling the REF submission process, and examples of good and bad practice. If you are interested in contributing, please contact ucurefwatch@ucu.org.uk (contributions can be made anonymously if preferred).

 

Leave a Reply

Your email address will not be published. Required fields are marked *