When the Research Excellence Framework was first proposed in 2007, as a replacement for the Research Assessment Exercise, one of its stated aims was to ‘avoid creating any undesirable behavioural incentives’. Yet anyone familiar with the way that universities have approached making their submissions to the first round at the end of 2013 knows this aim is not being realised. In fact quite the opposite is happening as many academics have found their work does not make the cut.
The REF is actually a powerful incentive scheme that sets the rules of the game institutions must play. It is not merely a survey of research done in universities as the RAE was intended to be (although it too led to game playing on an epic scale). Terminology is important. The REF is a `framework’ – which means a permanent incentive structure intended to shape the sector – and is completely different from an ‘exercise’ – an ad hoc investigation to find out what research otherwise autonomous institutions have been doing. So the question is whether these behavioural incentives produce results that are undesirable.
Another of the REF’s stated aims is ‘to provide a stable framework for [funding councils’] continuing support of a world-leading research base within HE’. Again, whether it brings forth world-leading excellence or mediocrity depends on how the incentives affect behaviour.
So the REF puts institutions in the position of having to behave tactically in order to maximise their grade profile. The key tactic they must play in this game is to decide which academics will have their work submitted for consideration by the panels and which excluded. They have to do that by second-guessing the judgments the panels are likely to make.
The problem is that there are no clear rules to guide them. Although the panels all published their guidelines in general terms, these are seriously lacking in crucial detail. So institutions have had to take these decisions in circumstances of fundamental uncertainty about the criteria the panels will actually apply in practice – but at the same time where the likely costs thought to follow from wrong decisions are pretty clear. HEFCE and the other funding councils have stated that research rated at two-star or less will not attract any funding from them. So there is a strong disincentive to submit work that could be rated as only (in REF-speak) ‘recognised internationally’ rather than ‘internationally excellent … but which falls short of the highest standards of excellence’ – a fine distinction but one that has crucial consequences.
What seems to be increasingly clear is that the REF incentives mean that some institutions see the reputational costs resulting from mistakenly submitting research that fails to receive the highest three- and four-star grades as very significant, much greater than those of mistakenly excluding that which could be rated as internationally excellent.
So they play safe and adopt what they call a ‘robust’ approach: ‘if in doubt, exclude it’. But this means that an unintended consequence of the REF is to marginalise and trivialise much new research, some of which may well be of greater value in the long term if given the support it deserves. The incentives are the wrong way round. They are failing to encourage new ideas.
If the REF is not to blight the very wellsprings of UK research, the incentive framework ought to make it in universities’ interests to be open minded enough to welcome new thinking – the radically original, the surprisingly innovative, the reconsideration of what was previously neglected, the rediscovery of what was previously overlooked, the unconventional, the opportunistic, the serendipitous, the odd – that we know from experience and history is such an essential part of what it means to be a world-leading research community.
One of the saddest things one increasingly hears in universities today is a head of department or senior administrator say, when presented with a novel research proposal, “where are the four-star papers in this?” before shelving it when such cannot be instantly pointed to. And we are finding that whole research programmes, whole fields, are being marginalised on the grounds that someone senior thinks they are “two-star” or “not-reffable”. This ought to be setting alarm bells ringing.
The current REF rules institutionalise perverse incentives whose long term effects on the UK higher education sector, currently one of the best in the world, will be dire, because it will stifle research. The REF is not a framework for research excellence.
Dennis Leech, University of Warwick