Flaws in institutional REF selection practices and their detrimental consequences

Both the UCU REF survey, and analysis of codes of practice on REF selection issued by individual HEIs indicate a number of deficiencies in institutional procedures established to select which outputs and individual researchers are to be submitted to the REF.  This is a particular concern for UCU given the link between REF performance and future career development for academic staff.

All HEIs intending to make a submission to the 2014 REF were required to publish a code of practice on the fair and transparent selection of staff for the REF submission. The official REF guidance produced by the joint funding councils’ REF team identifies four key principles – transparency, consistency, accountability and inclusivity – that these codes of practice should apply (see  http://www.ref.ac.uk/pubs/2011-02/ )

As well as adhering to the public sector equality duty as set out in the Equality Act (2010), these principles entail the development of a selection framework that is clearly communicated to staff, and which outlines consistent, inclusive and transparent procedures for selecting staff for the submission, with individuals and bodies involved in making the selection clearly identified.

HEIs were required to send their codes of practice to the REF team by the end of July 2012, so that they could be reviewed by the REF Equality and Diversity Advisory Panel (EDAP), charged with advising the UK funding councils on the adherence of the institutional codes of practice to the official guidance on submissions produced by the REF team. 

Following the 31 July deadline, UCU wrote to the REF team to express concern that a number of draft institutional codes that we had seen fell short of the requirements set out in the official guidance. Following its review of the 159 HEI codes of practice submitted, EDAP published a report on good practice in the codes (see http://www.ref.ac.uk/pubs/refcodesofpracticegoodpracticereport/#d.en.75885 ), noting that nearly half of the codes fell short of meeting some aspect of the guidance, and some of the codes required substantial revision.

Where EDAP found that the codes fell short of the guidance, the relevant funding body has corresponded directly with the institution concerned, and requested that they revise their code of practice accordingly in respect of the shortcomings identified. 

Nevertheless, reports from UCU branches and responses to our survey suggest that there remains a high level of dissatisfaction with regard to the lack of transparency and accountability in institutional REF selection processes. Most respondents did not regard institutional selection processes as transparent or easy to understand. There were also high levels of concern regarding the clarity with which decisions and evaluations were communicated, if communicated at all. There were significant levels of scepticism as regards the degree of expertise and training of those involved in the decision-making process, possible bias in decision-making, and the favouring of certain types of research over others. Many comments to the survey referred to the discriminatory nature of the REF, not just in terms of protected equality characteristics, but also in terms of types of research and research fields.

Analysis of codes of practice on REF selection issued by HEIs, and reports from UCU branches also suggest that there is a great deal of inconsistency within and between institutions as regards procedures and criteria used for selecting staff for the REF submission.  Concerns have been raised about the way in which the REF process in general, and institutional procedures in particular, give power to give certain managers in HEIs to make decisions which can make and break academic careers. Processes in some HEIs appear to allow certain individuals a great deal of discretion in making these decisions. 

There also appears to be a lack of confidence in appeal processes established by institutions in relation to REF inclusion decisions. The official REF guidance states that institutions should put in place appropriate and timely procedures to allow members of staff to appeal REF submission decisions. These procedures should allow enough time for staff to appeal after the decision, and for that appeal to be considered by the HEI before the final selection is made. The individuals handling the appeals should be separate to and independent of the individuals involved in the initial decision on selection. The EDAP report on good practice in the codes noted that this was an area where a number of Codes of Practice fell short of the guidance, for example by establishing processes that lacked sufficient independence, or where there was no assurance that appeal outcomes could be implemented before the submission deadline.

It appears from survey responses and numerous comments from UCU members that many institutions are still basing their decision-making on journal rankings or prestige (and in some cases, citation data), and favouring journal articles over other kind of outputs. This is despite the official funding councils’ REF guidance stressing the ‘underpinning principle’ that all types of research and all forms of research outputs across all disciplines will be assessed on a fair and equal basis, and that no REF sub-panel will make use of journal impact factors, rankings or lists, or the perceived standing of the publisher, in assessing the quality of research outputs. The guidance also recommends that HEIs do not use citation data to inform their submission decision (citation data will be used by a limited number of panels and only as ‘additional information’).  

Given the flawed nature of the REF selection process at HEI level, the REF cannot be regarded as an accurate indicator of the quality of academic research in the sector, and certainly not an accurate indicator of the quality of work of any individual academic.  Given that some HEIs are threatening career detriment and punitive sanctions for staff not included in the REF submission, the deficiencies in local selection processes and the wider flaws in the assessment framework, are of serious concern to UCU.  Hence our call for HEIs to issue assurances on no detriment for those staff not included in the REF.  Unfortunately, and with a few exceptions, such assurances have not been forthcoming at a number of HEIs. Institutional selection procedures and detrimental processes arising from them will therefore need to be monitored closely.

In this section of the blog, we are seeking to gather together commentaries and case studies on the way in which HEIs are handling the REF submission process, and examples of good and bad practice, including examples of detrimental processes that have arisen in relation to individuals not included in the REF submission. If you are interested in contributing, please contact ucurefwatch@ucu.org.uk (contributions can be made anonymously if preferred).


Leave a Reply

Your email address will not be published. Required fields are marked *