The Research Excellence Framework (REF) is the latest in a series of 6-yearly attempts to assess the quality of research in UK universities. It’s used to decide how to allocate about £1.6 billion per year of taxpayers’ money, the so-called "quality-related" (QR) allocation.
It could have been done a lot worse. One of the best ideas was that only four papers could be submitted, whatever the size of a research group. After much argument, the judgment panels were told not to use journal impact factors as a proxy for quality (or, for lack of quality), though it’s clear that many people did not believe that this would be obeyed, But it cost at least £60 million. At UCL alone, it took 50 – 75 person-years of work. and the papers that were submitted were assessed by people who often would have no deep knowledge about the field, It was a shocking waste of time and money, and its judgements in the end were much the same as last time.
Did the REF benefit science?
It’s frequently said that the REF improved the UK’s science output. The people who claim this need a course in the critical assessment of evidence. Firstly, there is no reason to think that science has improved in quality in the last 6 years, and secondly any changes that might have occurred are hopelessly confounded with the passage of time, the richest source of false correlations.
I’d argue that the REF has harmed science by encouraging the perverse incentives that have done so much to corrupt academia. The REF, and all the other university rankings produced by journalists, are taken far too seriously by vice-chancellors and that does active harm. As one academic put it
"This isn’t about science – it’s about bragging rights, or institutional willy-waving. "
There are now serious worries about lack of reproducibility of published work, waste of money spent on unreliable studies, publication of too many small under-powered studies, bad statistical practice (like ignoring the false discovery rate), and about exaggerated claims by journals, university PR people and authors themselves. These result in no small part from the culture of metrics and the mismeasurement of science. The REF has added to the pressures.
It is highly unsatisfactory, so the only real question becomes what should be done instead?
What’s to be done?
Transferring all the QR money to Research Councils won’t work. It would merely encourage the grossly bad behaviour that we’ve seen at Imperial College London, Warwick University, Kings College London and Queen Mary College London, all of whom have fired successful senior staff simple because their grant income wasn’t deemed big enough. (This is odd because the same managers whine continually that they make a loss on research grants, but that’s another question.) It’s been suggested that this could be avoided by reducing considerably the overheads that come with grants, but this would leave a shortfall that, without QR, would be impossible to make up.
At present a HEFCE working group is considering the possibility that metrics might be used in the next REF. It’s a sensible group of people, and they are well aware of the corrupting influence of metrics, and the lack of evidence that they measure the quality of research. So if reading papers takes too much time and money, and metrics are likely to lead to widespread "gaming" (a euphemism for cheating), what should be done?
I made a suggestion in 2010, but it seems to have been totally ignored, despite appearing in the Times (in their premier
Thunderer opinion column. So I’ll try to make the case again, in the context of the REF.
A complete re-thinking of tertiary education is needed,
Proposal for a two stage higher education system
It seems to be a good thing that such a large proportion of the population now get higher education. But the university system has failed to change to cope with the huge increase in the number of students.
The system of highly specialist honours degrees might have been adequate when 5% of the population did degrees, but that system seems quite inappropriate when 50% are doing them.
There are barely enough university teachers who are qualified to teach specialist 3rd year or postgraduate courses. And many teachers must have suffered from (in my field) trying to teach the subtleties of the exponential probability density function to a huge third year class, most of whom have already decided that they want to be bankers or estate agents.
These considerations have driven me to conclude, somewhat reluctantly, that the whole system needs to be altered.
Honours degrees were intended as a prelude to research and 50% of the population are not going to do research (fortunately for the economy). Vice-chancellors have insisted on imposing on large numbers of undergraduates, highly specialist degrees which are not what they want or need.
I believe that all first degrees should be ordinary degrees, and these should be less specialist than now. Some institutions would specialise in teaching such degrees, others would become predominantly postgraduate institutions, which would have the time. money and expertise to do proper advanced teaching, rather than the advanced Powerpoint courses that dominate what passes for Graduate Schools in the UK.
There would, of course, be almighty rows about which universities would be re-allocated to teach ordinary degrees. That’s not a reason to educate students in 2015 using a pre-war system.
The two-stage system would be more egalitarian than the present one
I anticipate that some people might think that this system is a reversion to the pre-1992 divide between polytechnics and universities. It isn’t. The pre-1992 system labelled you as either polytechnic or university: it was a two-tier system. I’m proposing a two stage system. The two sorts of institution work in series, not in parallel.
Such a system would be more egalitarian than now, not less.
Everyone would start out with the same broad undergraduate education, and the decision about whether to specialise, and the area in which to specialise, would not have to be made before leaving (high) school, as now, but would be postponed until two or three years later. That’s a lot better, especially for people from poorer backgrounds.
If this were done, most research would be done in the postgraduate institutions. Of course there are some good researchers in institutions that would become essentially teaching-only, so there would have to be chances for such people to move to postgraduate universities, and for some people to move in the other direction.
This procedure would, no doubt, result in a reduction in the huge number of papers that are published (but read by nobody). That is another advantage of my proposal. It’s commonly believed that there is a large amount of research that is either trivial or wrong. In biomedical research, it’s been estimated that 85% of resources are wasted (Macleod et al., 2014).
It’s well-known that any paper, however bad, can be published in a peer-reviewed journal. Pubmed, amazingly, indexes something like 30 jouranls devoted to quack medicine, in which papers by quacks are peer-reviewed by other quacks, and which are then solemnly counted by bean-counters as though they were real research. The pressure to publish when you have nothing to say is one of the perverse incentives of the metrics culture.
It seems likely that standards of research in second-stage universities would be at least as high as at present. It that’s the case then QR could simply be allocated on the basis of the number of people in a department. Dorothy Bishop has shown that even under the present system, the amount of QR money received is strongly correlated with the size of the department (correlation coefficient = 0.995 for psychology/neuroscience).
From Dorothy Bishop’s blog. r = 0.995
Using metrics produces only a tiny increase in the correlation coefficient for RAE data. It could hardly be any higher than 0.995
In other words, after all the huge amount of time, effort and money that’s been put into assessment of research, every submitted researcher ends up getting much the same amount of money.
That system wouldn’t work at the moment, because, sadly, universities would, no doubt, submit the departmental cat for a share of the cash. But it could work under a system such as I’ve described. The allocation of QR would take microseconds and cost nothing.
How much would the two-stage system cost?
To have any hope of being accepted by politicians, the two-stage system would probably have to cost no more than the existing system. As far as I know. nobody seems to have made any serious attempt to work out the costs. Perhaps they should. It won’t be easy because an important element of the two-stage system is to improve postgraduate education, and postgraduate education was forgotten in the government’s "reforms"
Much would depend on whether the first stage, ordinary degrees could be taught in two years. In an institution that does little research, there would be no justification for the long summer vacation. Something comparable with (high) school holidays would be more appropriate, and if a decent job could be done in two years, that could save enough money to pay for the rest. It would also mimimise the debt that hangs round the neck of graduates.
The cost of running the second stage would depend on how many students opted (and qualified) to carry on to do an honours degree, and on how many of those wanted go on to graduate school and higher degrees. The numbers of people that went on to specialist honours degrees would inevitably be smaller than now, so their education would be cheaper. But, crucially, they could be educated better. And because of the specialist researchers in a postgraduate institution, it would be possible to have real postgraduate education in advanced research methods,
At present, Graduate Schools in the UK (unlike those in the USA) rarely teach topics beyond advanced Powerpoint, and that’s a recipe for later mediocrity.
In order to estimate the actual cost, we’d need to know how many people wanted to go beyond the first degree (and qualified to do so). If this were not to large, the proposed system could well be cheaper than the presnet one, as well as being more egalitarian, and providing better postgraduate education. The Treasury should like that.
The California System
It will not have escaped the readers’ attention that the two stage system proposed here has much in common with higher education in the USA. In particular, it resembles the University of California system, which was started in 1960. It became a model for the rest of the world.
Meanwhile, the UK persists with a pre-war system of specialist honours degrees that is essentially unchanged since only a handful of people went to universities.
It’s time for the UK to have a serious debate about whether we need to change.
I just noticed this, from the inimitable Laurie Taylor. It is dated 4 July 2013. Who says the REF does not encourage cheating?
Are you a distinguished academic researcher looking to supplement your income? Then look no further. Poppleton is offering 24 extraordinarily well-paid and extraordinarily part-time posts to leading scholars in almost any discipline who will help to raise its profile in the research excellence framework.
These posts will follow what is known as the Cardiff-Swansea paradigm in that successful candidates need not have conducted any of their distinguished research at Poppleton, have no need to ever visit the actual campus, and can be assured that their part-time contracts will expire immediately after the date of the REF census.
Please apply marking your application “REF FARCE”.
3 February 2015
The day after this post appeared the Guardian published a version of it which discussed only the two-stage degree proposals but omits the bit about the Research Excellence Framework (REF 2014). The title was "Honours degrees aren’t for all – some unis should only teach two-year courses". There are a lot more comments there than than here.. I assume that the headline was written by one of those pesky subeditors who failed to understand what’s important (the two year degrees were just a suggestion, nothing to do with the main proposals).
3 April 2015
As an experiment, this blog has been re-posted on the Winnower. The advantage of this is that it now has a digital object identifier, DOI: 10.15200/winn.142809.94999
*The problem is the idea of 50% of school-leavers being able to do courses which used to require the abilities only found in the top 10%. The broader political argument is about elitism – I mean we could all be neuro-surgeons if we wanted? That’s the myth perpetuated by American educational dogma. The truth is the extension of university places has resulted in an inevitable dumbing-down of degree courses. It’s not what employers want – they constantly complain of “graduates” who can’t spell, do sums, follow instructions of think logically about problem-solving. There was more academic rigour in ‘A’ levels of the 1970s than there is in an Honours degree course now. More and more people do post-grad degrees – not because they are particularly interested in research – but as a way of distinguishing themselves from the hordes with 1st degrees. Adding another layer of tertiary education is not the solution – it doesn’t deal with how meaningless honours degrees have become. Furthermore, it is wrong to perpetuate a commercial education system where young people acquire debts that many will struggle to ever pay for, (but without a dgree they can’t get a job) when the problems of paying a mortgage and pension contributions is already near unaffordable.
I don’t agree at all with what you say. I’m very much in favour of as many people as possible getting some sort of higher education. Education is good.
Your comment sounds dangerously close to nostalgia for a past golden age that never existed. I recall that when I graduated (in 1960) most of my class in biological sciences got decent degrees without ever learning calculus, I doubt that many of them could have written down the equation for exponential decay. The same is true now.
I agree that the proportion of semi-competent graduates has gone up, but what matters more is that the number of good graduates has increased.
I think it is badly wrong to say that there “was more academic rigour in ‘A’ levels of the 1970s than there is in an Honours degree course now”.
My proposal doesn’t amount to “another layer of tertiary education”. It might be possible in the same time as a standard 3 year degree, and would certainly take no longer than the standard 4 year degree in Scotland.
I do suspect that the rigour of third year classes has fallen a bit if only because you lose a point on student assessment for every equation you include, and it’s true that very few 3rd class degrees are given now. Insofar as that’s a problem, it’s a problem that the 2-stage degree would go a long way to solving
Because not everyone would go on to the honours degree it would be possible to impose higher standards for the latter, as well a proper advanced teaching in graduate schools for people who got that far. So everybody wins.
There are a lot more comments at the Guardian Higher Education, which published the two-stage degree part of this blog (but not the comments on the REF).
*David – I was educated in the British system through my B.Sc. (Exon), and then came to the US 36 years ago, to teach Mathematics. Most biology degrees here do not require any Calculus, and most degrees are indeed very watered-down., especially in Mathematics, English and Science.
Unfortunately it is not the case that “every submitted researcher ends up getting much the same amount of money”. The correlation plot is quite misleading. As is all to frequently the case for such plots, it is dominated by a small number of cases that contribute large values to the correlation coefficient. The plot only actually shows that most of the money goes to a few large institutions and that those large institutions are judged to have rather similar proportions of high quality research. Perhaps this is clearer, if using the same data (for psychology, neuroscience and related submissions) we plot relative funding per head (based on the assumption that only 4* research will be funded) directly vs. size of submission.
It isn’t clear what degrees you are talking about. My experience in teaching in Yale’s Graduate School was that people were taught far more difficult stuff than they encounter in the UK.
The two-groups-of-points artefact is indeed a very common reason for misleading correlation coefficients. It’s apparent in your graph, but not in Dorothy Bishop’s which shows an essentially linear relationship over a wide range. Your graph is very non-linear and so not suitable for calculating a correlation coefficient at all.
It was my understanding that 3* research would get funding too. Is that know yet?
Have you got links to your data source? It would be interesting to attach names to the points.
All the REF outcomes are at http://results.ref.ac.uk/DownloadResults in an Excel spreadsheet that one can play with to one’s haerts content/despair.
Part of the point of Dorothy Bishop’s blog was inspired by the strong rumour (doing the rounds of VCs) that Treasury is minded to only fund “world class” research in future (why should it waste its money on the merely internationally excellent). Dorothy used her graph to suggest this wouldn’t make much difference from the current system ( (whereby 4* research gets 3 times the money as 3* research). Obviously the non-linearity means that conclusion would only be true for about 20 (mostly larger) institutions, everyone else would lose out.
A reason for thinking this may be a rumour with some truth is that the widespread increase in scores would mean that a greater fraction of the pot under the current formula would go to smaller institutions with some (say 30-50%) internationally excellent research (but not much world class); every time this scenario has happened in the past the funding formula has been changed to make sure the larger institutions don’t lose money.
@David – I would hesitate to use Yale as an example. They are arguably much more selective than Oxbridge, and their typical student is most decidedly not representative of even the median US undergraduate.