Perverse incentives mean the REF encourages mediocrity rather than excellence

When the Research Excellence Framework was first proposed in 2007, as a replacement for the Research Assessment Exercise, one of its stated aims was to ‘avoid creating any undesirable behavioural incentives’. Yet anyone familiar with the way that universities have approached making their submissions to the first round at the end of 2013 knows this aim is not being realised. In fact quite the opposite is happening as many academics have found their work does not make the cut.

The REF is actually a powerful incentive scheme that sets the rules of the game institutions must play. It is not merely a survey of research done in universities as the RAE was intended to be (although it too led to game playing on an epic scale). Terminology is important. The REF is a `framework’ – which means a permanent incentive structure intended to shape the sector – and is completely different from an ‘exercise’ – an ad hoc investigation to find out what research otherwise autonomous institutions have been doing. So the question is whether these behavioural incentives produce results that are undesirable.

Another of the REF’s stated aims is ‘to provide a stable framework for [funding councils’] continuing support of a world-leading research base within HE’. Again, whether it brings forth world-leading excellence or mediocrity depends on how the incentives affect behaviour.

So the REF puts institutions in the position of having to behave tactically in order to maximise their grade profile. The key tactic they must play in this game is to decide which academics will have their work submitted for consideration by the panels and which excluded. They have to do that by second-guessing the judgments the panels are likely to make.

The problem is that there are no clear rules to guide them. Although the panels all published their guidelines in general terms, these are seriously lacking in crucial detail. So institutions have had to take these decisions in circumstances of fundamental uncertainty about the criteria the panels will actually apply in practice – but at the same time where the likely costs thought to follow from wrong decisions are pretty clear. HEFCE and the other funding councils have stated that research rated at two-star or less will not attract any funding from them. So there is a strong disincentive to submit work that could be rated as only (in REF-speak) ‘recognised internationally’ rather than ‘internationally excellent … but which falls short of the highest standards of excellence’ – a fine distinction but one that has crucial consequences.

What seems to be increasingly clear is that the REF incentives mean that some institutions see the reputational costs resulting from mistakenly submitting research that fails to receive the highest three- and four-star grades as very significant, much greater than those of mistakenly excluding that which could be rated as internationally excellent.

So they play safe and adopt what they call a ‘robust’ approach:  ‘if in doubt, exclude it’. But this means that an unintended consequence of the REF is to marginalise and trivialise much new research, some of which may well be of greater value in the long term if given the support it deserves. The incentives are the wrong way round. They are failing to encourage new ideas.

If the REF is not to blight the very wellsprings of UK research, the incentive framework ought to make it in universities’ interests to be open minded enough to welcome new thinking – the radically original, the surprisingly innovative, the reconsideration of what was previously neglected, the rediscovery of what was previously overlooked, the unconventional, the opportunistic, the serendipitous, the odd – that we know from experience and history is such an essential part of what it means to be a world-leading research community.

One of the saddest things one increasingly hears in universities today is a head of department or senior administrator say, when presented with a novel research proposal, “where are the four-star papers in this?” before shelving it when such cannot be instantly pointed to. And we are finding that whole research programmes, whole fields, are being marginalised on the grounds that someone senior thinks they are “two-star” or “not-reffable”. This ought to be setting alarm bells ringing.

The current REF rules institutionalise perverse incentives whose long term effects on the UK higher education sector, currently one of the best in the world, will be dire, because it will stifle research. The REF is not a framework for research excellence.

Dennis Leech, University of Warwick

  1 comment for “Perverse incentives mean the REF encourages mediocrity rather than excellence

  1. Ryan
    30 December 2013 at 9:03 pm

    Our externals rated my REF at an average of 3*, so it was submitted: 3 peer reviewed articles and a book. And for over two years I went to steady REF meetings every 4-6 months where you had to reflect on your submission with a panel of colleagues. Then you’re subjected to endless internal and external review before a decision is made as to your ‘star’ ranking and if you’ll be included. Its not as stodgy and petty as Leech suggests, at least not in my experience. I get the gist of this article, but my research was pretty ‘original’ and ‘innovative’, as was most of the other submissions in my department. Leech is an economist at Warwick, and in his discipline and institution, versus mine, I would guess that novel and innovative work is harder to produce. Also, he may have a grievance because his work, or work he thinks that is being excluded is probably not published at very well know, high impact journals or publishers. But, I argue, if the work is so fantastic, 3 star or 4 star WHY was it NOT publish it at a top journal or publisher? I’ve heard a number of academics complain that their article at a small non-peer reviewed journal was not being considered, and this was not fair. Well, if its such a great piece of scholarship, why did you not, or why was is not picked up at a top journal or publisher where it would get the widest exposure? I think this entire article is superfluous to the real problems of the REF however. Which are: how the hell do you actually put together an impact study that is a reliable indicator of ‘impact’. And why is teaching, measured in awards and other indicators, not included along with publications and ‘impact’?

Leave a Reply

Your email address will not be published. Required fields are marked *