Vince Cable, the Secretary of State for Business, Innovation and Skills, said on the Radio 4 Today programme on September 8th 2010
“There was some estimate on the basis of surveys done recently that something in the order of 45 per cent of the research grants that were going through was to research that was not of excellent standard. So the bar will have to be raised.”
The suggestion that 45 percent of research is mediocre provoked a storm, first on Twitter, than in blogs. One of the earlier blogs was In one day, Vince Cable has become an object of ridicule and loathing Those that followed were scarcely more flattering. The number he quoted was simply wrong.
Unravelling Cable says "when the text of the speech was released, I was shocked by what it revealed about the Secretary of State’s grasp of his brief.".
A legion of people have tried to decode what he meant. The purpose of this post is to go a bit further, to investigate the problem of mediocre research and to suggest a change of policy that might help.
This appears to be what Cable should have said.
(1) His comments don’t refer to the main source of money for research at all. They refer to "quality-related" (QR) money given to universities by the Higher Education Finding Council. It is intended to support the infrastructure for support, but it vanishes into the ever-expanding administration and most researchers don’t see a penny of it.
(2) QR money is not given to individual researchers to do research, it is given to the university retrospectively, on the basis of the score in a vast, time-consuming, assessment known as the Research Assessment exercise. This grades departments on the basis of the amount of grade 1 2 3 or 4 research they do.
(3) Cable’s comment . on the BBC Radio 4 Today Programme that “45% of research grants were not of excellent standard", caused uproar, and rightly so, because it shows a total lack of grasp of how science is funded, The "45%" doesn’t refer to grants at all, but it is the percentage of work that was judged in the RAE to be grade 1 or 2 (less good) rather than 3 or 4 (very good).
(4) Work that is graded 2 gets little QR money, and 1 gets none at all. David Sweeney, a HEFCE bureaucrat not generally noted for his understanding of research, points out that the 45 percent of 1 and 2 work gets only 7 percent of the funds. Using Cable’s criterion his number should have been 7% not 45%.
(5) In physics at least, the RAE panel claims it was told to use norm-referencing. This means that they are told roughly what fractions 3 and 4 grades to produce. HEFCE deny this is the case, but it is quite usual for big organisations to lie about this sort of thing. Insofar as norm-referencing was imposed, the fraction of research that is labelled mediocre is pre-determined, and is quite independent of quality. It means no more than saying that half the people are below average. It is just a statistical inevitability (if the distribution is symmetrical). It tells you absolutely nothing about the quality of research,
(6) A figure far more important than any of these is that only 10 – 20% of research grant applications get funded. It takes a long time to write a research grant application, something like two months. That is a major time-consuming activity for scientists, who should be thinking about science and doing experiments. Around 85% of that effort is fruitless. The cost in salaries and lost output of writing grants that fail is enormous. Being high alpha rated is certainly no guarantee of getting funds. That is the number that Cable should have produced, but didn’t.
So what did Cable get right, and why?
This is the bit that hasn’t been discussed much in the comments so far.
If it so hard to get a grant, why is there a widespread perception that quite a lot of published work is, if not wrong, at least trivial?
(1) Most work has always been trivial. Great breakthroughs are very rare events. But let’s accept, for the sake of argument, that the widespread view that it is worse now that in used to be, or at least that quality hasn’t improved.
(2) There is now enormously more research than before. That means more top-rate work, but, perhaps, even more bad or trivial work
(3) The Research Assessment Exercise (RAE) is on of the main reasons why the fraction of good papers is perceived to have decreased. It has probably done more harm to research standards than any other single change in my lifetime.
(4) Although it is tempting to blame politicians for the harm done by the RAE, the greatest harm is actually done by senior academics and other ex-researchers, and their numerous administrative hangers-on, who apply intolerable pressure on their juniors to publish large numbers of papers, preferably in one of three ‘top’ journals. This inevitably leads to large numbers of trivial papers being produced, because people think they won’t be promoted if the don’t go along with the senior bullies,
(5) The fate of young researchers is made even worse by HR’s attempts at "well-being" (major post coming up on that topic), the money wasted on ‘Roberts agenda’ skills training and the utterly vacuous Concordat
(6) A consequence of this sort of pressure is that anyone who wants to think deeply, or to understand properly, the basis of what they are doing, is quite likely to be fired for lack of "productivity" before they produce their best work (I speak here of biomedical sciences. I presume that physicists can’t get way so easily with poor understanding of principles of their subject).
(7) The result is a system that is not just over-competitive, but positively cruel to young scientists. The miracle to me is that anyone wants to work in science at all in the present state of universities. It is a sign, I guess, of just how wonderful it is to find something new, that people still put up with a system that seems, at times, not much different from slavery. See, for example, The Mismeasurement of Science by Peter A. Lawrence
(8) Not all the slavery is, of course, quite was bad as the famous chemist at Caltech who berated his slaves for not working (for his glory) seven days a week (though things not far short of this are quite common). And not all universities are quite so stupid as the Texas A&M University, which is reported to be thinking of hiring on the basis of the amount of grant income you bring in, although Imperial College London got alarmingly close to this sort of insanity. I guess I shouldn’t feel bad about other universities behaving in a way that makes anyone who is any good not want to work for them, but the matter is too important for one to worry about inter-university rivalry.
(9) The over-competitiveness and encouragement of trivial science, quantity rather than quality, has been going on for long enough now, that people who have risen on that basis now have power, and are to be found even on review panels of grant-giving bodies.
(10) Organisations like the Medical Research Council used to have permanent staff who developed a high level of expertise in the subjects they dealt with, and a great deal of expertise in the critical duties of knowing which referees to select, and how to judge what they wrote. More recently, the turnover of MRC staff has been too great for that sort of expertise to be well-developed, I have no axe to grind myself. My last program grant as PI (1999 – 2004) was funded, as was its successor (2004 – 2009, in which I was co-applicant). But recently I have seen feedback on failed grant applications (not mine) that suggest that the review panel either hadn’t read or hadn’t understood them.
(11) There is an enormous shortage of money for ‘response-mode’ grants. That means grants submitted by individuals to fund projects that the individual thinks will work. One reason for that is the research councils and charities have, increasingly, ring-fenced funds for work in a particular area, which some committee has decided ie important. Often this results in money being given to projects that don’t work very well (as I have seen at first hand when on the panel for the BBSCR Neuron initiative). These "initiatives" may sound good on paper to politicians, but they result in mis-spending of taxpayers’ money.
(12) One thing that Cable is dead right about ir that the ‘graduate tax’ is by far the fairest way to fund degrees. Sadly vice-chancellors line up to condemn it (you can’t get the staff these days).
So what can be done?
I’ve listed a lot of criticisms, but what can be done about it?
I can see a couple of things that could be done. The main thing is to reduce the intense competitiveness that leads to low quality. The competitiveness arises in part because of the large increase in the number of universities that took place in 1992, when the then Conservative government converted at a stroke polytechnics and technical colleges into universities, This was done largely to increase the number of undergraduate students, something of which I advocate strongly. I also feel strongly that teaching at an advanced level should be done by people who are doing research in the area they are teaching about. This is what governments have tried to do since 1992, but the numbers just don’t add up. There are simply not enough good researchers to teach half the population, yet the promotion of everyone has been made to depend largely on research.
One way would be to retain honours degrees but make the post-1992 universities into teaching only institutions. That would be the wrong solution in my view. It would result in a lot of teaching being done by people barely able to cope with advanced stuff.
(1) The conservatism of some senior academics has meant that they have failed to recognise that the traditional honours degree is quite unsuitable for a mass education system in which 50% of people do a degree.
(2) We should abandon altogether the honours degree system, which attempts in 3 years to take people from high school level to research level in 3 years (even with smaller numbers of people it often failed to do that anyway).
(3) We should start with much wider general degrees where teaching could be at a lower level and be done in universities that did little research. Such degrees would still aim at critical thinking rather than being purely vocational. Reluctant though I am to see teaching and research separated, it has become an economic necessity and the harm should not be too great if the separation applies only to general first degrees,
(4) After this general first degree, students would either do vocational training, or if they wished to continue along the academic line, they would go to graduate school.
(5) By graduate school, I mean teaching in the advanced aspects of their chosen subject, as is done in the USA. Most UK universities now have something called a graduate school, but they are largely charades which teach advanced powerpoint presentation but nothing intellectual. Our own summer school was originally taught as part of the UCL graduate school, but was dropped by them on the grounds that it was education not training. Protests that a knowledge of mathematics was the ultimate transferable skill in science, fell on deaf ears.
(6) The graduate schools would be the place where the advanced teaching was done, and also where most research was done. To make this feasible for the staff, they would have little, or even no, undergraduate teaching. They’d be more like ‘institutes of advanced studies’.
(7) This proposed system is, of course, much more like the system in the USA than the present UK system. It’s worked rather well there. We should try it.
Your comments are spot on, and on the subject of most scientific work being trivial, I am reminded of Gandhi’s words that “Whatever you do will be insignificant, but it is very important that you do it.” Contrast this with Cable’s plans to cut the science budget, which are significant, and which it is important should not be done.
Remember Sturgeon’s (second) law: http://en.wikipedia.org/wiki/Sturgeon%27s_Law.
By all means let’s cut the standard undergraduate degree to two years: we lose money on them anyway. But please don’t let’s think that will solve the problem of mediocre research! In fact, the more people have undergraduate degrees, the less this will function as a signal of ability, and so the more people will stay on to do graduate degrees.
I like your last sentence of proposal (5) in particular. We have the same problem here, where all of the mathematically-illiterate administrators and politicians fail to understand that:
a) Trying to increase the number of science and engineering graduates requires more resources for mathematics, and
b) Learning mathematics appears to be an innate ability
I agree with everything you say here.
At first I was unsure about your support for putting large numbers of people through university as I am very against the previous Labour Government’s arbitrary target of getting 50% of the population a degree. However, run the way you describe it here, I can really see the value.
Certainly the population as a whole needs far better training in critical thinking!
That Gandhi quotation must be in the head of every blogger, every activist in any area.
I should have distinguished between trivial research and poor research. It is the latter that we need to worry about. There plenty of hyped up papers that don’t show what the authors claim they show. The proliferation of these is encouraged by the need to publish when you have nothing to say, in order for get promotion or to get a grant. Of course if the promotion committees, and the funding agencies did their jobs properly, they would spot this sort of behaviour, but only too often they don’t.
Trivial papers are quite a different matter, I have always resisted the temptation to say, when refereeing a paper, to say that it it should be rejected because it is boring. Experience has shown that what might appear boring at the time may, ten years later, turn out to provide a crucial clue. I suppose that more often the boring results are still boring ten years on, and they get forgotten. But nobody can say at the time which will turn out to be important later, so work that seems trivial, but well-conducted is an essential part of the system. Its existence is merely a reflection of the self-evident truth that research, by its nature, can never be 100 percent efficient. It is littered with false starts and blind alleys.
Thanks, an interesting read, I’d been hoping to find time to look into this a bit more myself, and am glad you have laid out your thoughts on this so well.
Having read and mulled over your proposals, I’m finding myself more and more in agreement with them; though I can’t comment much on the academia side of things. From my point, based in industry, I can also see the benefit of this approach.
However, after briefly mulling it over, I also find myself wondering if this kind of approach might end up producing poorer scientists/engineers, within industry, than having them from a more academic background. Sometimes it is the broadness of the education, at a higher level, that allows people to transfer knowledge from different areas to those currently under consideration.
This argument needs more thought (by me), and may just be spurious ill-considered nonesense tapped out in a lunch break…
Thanks for the comment. It’s valuable to have the view from somebody in industry.
There would be a lot of details to think through. but it seems to me that your objection would be overcome if the best graduates advanced to the graduate school stage, where they could be taught to a higher level than the present honours degree. There is no reason why people should have to go on to a PhD after a year of graduate school, if they wanted to go into the outer world at that stage. In fact I’d imagine that jobs with a technical content would require the graduate school year.
The problem at the moment is that substantial numbers of final year honours students have already decided that they don’t want to do anything technical or scientific at all, but to go into banking.
Actually, no, I think I agree with you. I think there are definite perks to such an approach.
I know I often lament my friends who left science to become chartered accountants or the like (and there are a quite a few). Often to their annoyance up the pub too…
As an aside, I’d also like to see the larger, and smaller really, companies taking on more staff without degrees under the current system.
There’s no reason why a sufficiently bright A-level student shouldn’t be able to come straight into the lab for some of the more routine work, and trained up from there, both in house, and through part-time (university) education. Companies could get the staff trained in exactly what they want, increase retention, while truly developing peoples education. Sans that (damn) student loan…
This is similar to something we have done over the last 5 or so years, almost a bio-analytical apprenticeship (though lacking a truly defined structure), and it’s working well for us. One has even gone off to join an academic lab here in Oxford as an RA (slightly tainting my retention argument…dammit!)
Though my suggestion to take on A-level students tends to be avoided at higher levels…and now we’ve been bought by a far larger company, their recruitment policies seem to negate this approach and the inertia to change is even harder to overcome. Which is a shame.
There’re arguments and many things to consider if this was on a larger scale of course, and would be a fairly radical approach for big pharma (for example) but, if more common, could be a good compliment to the current system, or your system outlined above.
Doubt it’d happen though.
(apologies for length)
[…] The article reproduced here is the original 800-word version of proposals made already on this blog. […]