In the course of thinking about metrics, I keep coming across cases of over-promoted research. An early case was “Why honey isn’t a wonder cough cure: more academic spin“. More recently, I noticed these examples.
“Effect of Vitamin E and Memantine on Functional Decline in Alzheimer Disease".(Spoiler -very little), published in the Journal of the American Medical Association. ”
and ” Primary Prevention of Cardiovascular Disease with a Mediterranean Diet” , in the New England Journal of Medicine (which had second highest altmetric score in 2013)
and "Sleep Drives Metabolite Clearance from the Adult Brain", published in Science
In all these cases, misleading press releases were issued by the journals themselves and by the universities. These were copied out by hard-pressed journalists and made headlines that were certainly not merited by the work. In the last three cases, hyped up tweets came from the journals. The responsibility for this hype must eventually rest with the authors. The last two papers came second and fourth in the list of highest altmetric scores for 2013
Here are to two more very recent examples. It seems that every time I check a highly tweeted paper, it turns out that it is very second rate. Both papers involve fMRI imaging, and since the infamous dead salmon paper, I’ve been a bit sceptical about them. But that is irrelevant to what follows.
Boost your memory with electricity
That was a popular headline at the end of August. It referred to a paper in Science magazine:
“Targeted enhancement of cortical-hippocampal brain networks and associative memory” (Wang, JX et al, Science, 29 August, 2014)
This study was promoted by the Northwestern University "Electric current to brain boosts memory". And Science tweeted along the same lines.
Science‘s link did not lead to the paper, but rather to a puff piece, "Rebooting memory with magnets". Again all the emphasis was on memory, with the usual entirely speculative stuff about helping Alzheimer’s disease. But the paper itself was behind Science‘s paywall. You couldn’t read it unless your employer subscribed to Science.
All the publicity led to much retweeting and a big altmetrics score. Given that the paper was not open access, it’s likely that most of the retweeters had not actually read the paper.
When you read the paper, you found that is mostly not about memory at all. It was mostly about fMRI. In fact the only reference to memory was in a subsection of Figure 4. This is the evidence.
That looks desperately unconvincing to me. The test of significance gives P = 0.043. In an underpowered study like this, the chance of this being a false discovery is probably at least 50%. A result like this means, at most, "worth another look". It does not begin to justify all the hype that surrounded the paper. The journal, the university’s PR department, and ultimately the authors, must bear the responsibility for the unjustified claims.
Science does not allow online comments following the paper, but there are now plenty of sites that do. NHS Choices did a fairly good job of putting the paper into perspective, though they failed to notice the statistical weakness. A commenter on PubPeer noted that Science had recently announced that it would tighten statistical standards. In this case, they failed. The age of post-publication peer review is already reaching maturity
Boost your memory with cocoa
"Enhancing dentate gyrus function with dietary flavanols improves cognition in older adults. Brickman et al., Nat Neurosci. 2014. doi: 10.1038/nn.3850.".
The journal helpfully lists no fewer that 89 news items related to this study. Mostly they were something like “Drinking cocoa could improve your memory” (Kat Lay, in The Times). Only a handful of the 89 reports spotted the many problems.
A puff piece from Columbia University’s PR department quoted the senior author, Dr Small, making the dramatic claim that
“If a participant had the memory of a typical 60-year-old at the beginning of the study, after three months that person on average had the memory of a typical 30- or 40-year-old.”
Like anything to do with diet, the paper immediately got circulated on Twitter. No doubt most of the people who retweeted the message had not read the (paywalled) paper. The links almost all led to inaccurate press accounts, not to the paper itself.
But some people actually read the paywalled paper and post-publication review soon kicked in. Pubmed Commons is a good site for that, because Pubmed is where a lot of people go for references. Hilda Bastian kicked off the comments there (her comment was picked out by Retraction Watch). Her conclusion was this.
"It’s good to see claims about dietary supplements tested. However, the results here rely on a chain of yet-to-be-validated assumptions that are still weakly supported at each point. In my opinion, the immodest title of this paper is not supported by its contents."
NHS Choices spotted most of the problems too, in "A mug of cocoa is not a cure for memory problems". And so did Ian Musgrave of the University of Adelaide who wrote "Most Disappointing Headline Ever (No, Chocolate Will Not Improve Your Memory)",
Here are some of the many problems.
- The paper was not about cocoa. Drinks containing 900 mg cocoa flavanols (as much as in about 25 chocolate bars) and 138 mg of (−)-epicatechin were compared with much lower amounts of these compounds
- The abstract, all that most people could read, said that subjects were given "high or low cocoa–containing diet for 3 months". Bit it wasn’t a test of cocoa: it was a test of a dietary "supplement".
- The sample was small (37ppeople altogether, split between four groups), and therefore under-powered for detection of the small effect that was expected (and observed)
- The authors declared the result to be "significant" but you had to hunt through the paper to discover that this meant P = 0.04 (hint -it’s 6 lines above Table 1). That means that there is around a 50% chance that it’s a false discovery.
- The test was short -only three months
- The test didn’t measure memory anyway. It measured reaction speed, They did test memory retention too, and there was no detectable improvement. This was not mentioned in the abstract, Neither was the fact that exercise had no detectable effect.
- The study was funded by the Mars bar company. They, like many others, are clearly looking for a niche in the huge "supplement" market,
The claims by the senior author, in a Columbia promotional video that the drink produced "an improvement in memory" and "an improvement in memory performance by two or three decades" seem to have a very thin basis indeed. As has the statement that "we don’t need a pharmaceutical agent" to ameliorate a natural process (aging). High doses of supplements are pharmaceutical agents.
To be fair, the senior author did say, in the Columbia press release, that "the findings need to be replicated in a larger study—which he and his team plan to do". But there is no hint of this in the paper itself, or in the title of the press release "Dietary Flavanols Reverse Age-Related Memory Decline". The time for all the publicity is surely after a well-powered study, not before it.
The high altmetrics score for this paper is yet another blow to the reputation of altmetrics.
One may well ask why Nature Neuroscience and the Columbia press office allowed such extravagant claims to be made on such a flimsy basis.
What’s going wrong?
These two papers have much in common. Elaborate imaging studies are accompanied by poor functional tests. All the hype focusses on the latter. These led me to the speculation ( In Pubmed Commons) that what actually happens is as follows.
- Authors do big imaging (fMRI) study.
- Glamour journal says coloured blobs are no longer enough and refuses to publish without functional information.
- Authors tag on a small human study.
- Paper gets published.
- Hyped up press releases issued that refer mostly to the add on.
- Journal and authors are happy.
- But science is not advanced.
It’s no wonder that Dorothy Bishop wrote "High-impact journals: where newsworthiness trumps methodology".
It’s time we forgot glamour journals. Publish open access on the web with open comments. Post-publication peer review is working
But boycott commercial publishers who charge large amounts for open access. It shouldn’t cost more than about £200, and more and more are essentially free (my latest will appear shortly in Royal Society Open Science).
Hilda Bastian has an excellent post about the dangers of reading only the abstract "Science in the Abstract: Don’t Judge a Study by its Cover"
4 November 2014
I was upbraided on Twitter by Euan Adie, founder of Almetric.com, because I didn’t click through the altmetric symbol to look at the citations "shouldn’t have to tell you to look at the underlying data David" and "you could have saved a lot of Google time". But when I did do that, all I found was a list of media reports and blogs -pretty much the same as Nature Neuroscience provides itself.
More interesting, I found that my blog wasn’t listed and neither was PubMed Commons. When I asked why, I was told "needs to regularly cite primary research. PubMed, PMC or repository links”. But this paper is behind a paywall. So I provide (possibly illegally) a copy of it, so anyone can verify my comments. The result is that altmetric’s dumb algorithms ignore it. In order to get counted you have to provide links that lead nowhere.
So here’s a link to the abstract (only) in Pubmed for the Science paper http://www.ncbi.nlm.nih.gov/pubmed/25170153 and here’s the link for the Nature Neuroscience paper http://www.ncbi.nlm.nih.gov/pubmed/25344629
It seems that altmetrics doesn’t even do the job that it claims to do very efficiently.
It worked. By later in the day, this blog was listed in both Nature‘s metrics section and by altmetrics. com. But comments on Pubmed Commons were still missing, That’s bad because it’s an excellent place for post-publications peer review.
It is not uncommon for bloggers to be critical of science reporting in the mainstream media. Now is our chance to do something constructive about it. If you have opinions about this, please leave them in the comments here, and/or email them to
Here are some of my own opinions, to get things going. Many programmes I haven’t seen/heard, so my selection may not be representative, but it is wide enough to include examples that are superb and examples of some that I think are not good enough.
There are two particular topics that are real problems for broadcasters. One is the whole area of alternative medicine and the ‘supplement’ industry. The other is anything to do with climate change. Both have formidable lobby groups which, to the inexperienced journalist, may sound like quite plausible scientists (some even have academic titles). Creationists can also be a problem. though not many programmes take them very seriously. Both quacks and climate deniers rarely have anything to say that is real science. They have different motivations. Examples are given below.
Many programmes are superb
David Attenbrough is an obvious example. His programmes can’t be bettered. The photography is breathtakingly beautiful and the science is always accurate. For me, they alone are worth the licence fee, and I don’t want the licence fee to be reduced. It helps that Attenborough knows the science so well. It also helps that most of the time the science isn’t very difficult and isn’t very controversial either.
There have been many other superb programmes. Steve Jones own 6-part TV series "In the blood" was a beautiful example. The fact that his comments are sought frequently by the BBC is greatly to their credit. Much depends on producers being sufficiently well-informed to know whom to ask.
More recently Brian Cox’s "Wonders" series has provided an excellent example of how science programmes can be made popular without being inaccurate,
Also excellent were Jim Al-Khalili’s Chemistry: A Volatile History and Michael Moseley’s Medical Mavericks.
Simon Singh has made consistently good programmes. His wonderful documentary on Fermat’s Last Theorem was a masterpiece.. He is a master at making programmes that make really difficult subjects accessible to the public, without making them misleading.
Tim Harford’s programm, More or Less has made a great contribution to public understanding of statistics.
Ben Goldacre‘s two part Radio 4 Programme, The Rise of the Lifestyle Nutritionists was a superb explanation of a contentious area.
It would be useful if all presenters of programmes with any scientific content had to listen to Harford or Goldacre. It might make them a bit more critical about the problem of causality that beset the observational epidemiology items that predominate among the items picked out from press releases by so many news programmes.
These programmes were so successful because they were made by people who know what they are talking about. They confirm the view that the best science programmes are made by scientists, not by journalists. There are exceptions of course. It could be argued that some of Robert Winston’s programmes have strayed too far from his area of expertise to reach the same high standards. And some journalists have produced excellent programmes. Two examples follow.
Geoff Watts has kept up a consistently high standard on Radio 4. from Science Now, through Medicine Now to Leading Edge, the standard has always been high. It is good straight science in its social context. He avoids controversies, for example his excellent programme about Charles Darwin does not include a creationist to provide (phony) ‘balance’.
After the demise of Medicine Now in 1998, Watts wrote in the BMJ thus.
In the early days of the programme you could have listened for several weeks in a row without hearing from patients. I used to defend this on the grounds that Medicine Now was there to talk about disease and its treatment, not the experience of disease and its treatment. I was wrong. To make that distinction is simply to parallel the fault for which doctors themselves take a deal of stick—being interested in the illness to the exclusion of the person who’s suffering from it. I was persuaded, reluctantly, to accept more lay voices, and I am now embarrassed that I didn’t sooner see the need for them.
The swing of the pendulum may push the whole patient experience thing too far: to a point at which it’s professional knowledge and objective analysis that is elbowed into the wings. One of the vogue concepts among BBC managers in recent years has been “accessibility”. But this is a weasel word, too easily used as justification for editing out anything that might require the audience to concentrate and think. If a patient has a rash, it’s a lot simpler for the reporter to inquire about the urge to scratch than to explore the events in the immune system that caused the skin to redden, swell, and itch in the first place. How sad if people lose an opportunity to hear from the researchers, speaking their own words, who are actually trying to find out.
This summarises a lot of the problems of science programmes. They too easily become trivial vox pops, and Watts resisted this tendency very successfully.
A great problem for programmes about medicine arises from the pressure exerted by the alternative medicine industry (ot which more later). Watts would not tolerate nonsense. He says
Medicine Now was stabled in the BBC’s Science Unit, and it was the broad acceptance of science and its methodology which shaped editorial choices. When complementary medicine was on the agenda, we expected evidence from our contributors not testaments of faith.
Material World is another good Radio 4 programme. Quentin Cooper does, on the whole, a good job. But sometimes even he falls foul of the phony balance argument After my piece in Nature on the shameful degrees in pseudo-scientific medicine got discussed on Material World, (audio here) but my opponent was not a scientist at all, but the head of “Complementary Therapies” at the University of Westminster, a man who presides over courses that teach “amethysts emit high Yin energy“. It is simply impossible to have a proper scientific discussion with people who believe nonsense like that. They don’t accept the ground rules at all. It is a good example of phony balance (see below).
Some programmes are quite bad
Alternative Medicine: the evidence. This series if three TV programmes was shown in February 2006 on BBC 2, in conjunction with the Open University. It illustrates well three problems with science programming. (1) Despite the title, tt was surprisingly weak at showing evidence, (2) It showed the defensive and unhelpful response that, only too often, the BBC shows when complaints are made. And (3) it showed that association with a university is not, per se, enough to guarantee quality.
Because of the title, I’d looked forward to this programme, and made minute by minute notes, which are recounted in BBC2 and the Open University on Alternative Medicine. It turned out that the evidence was thin on the ground, and what there was was not always accurate I complained to the BBC, but got nowhere [download my complaint and some subsequent correspondence]. I was fobbed off with defensive PR. (Much the same happened when I complained to the Open University.)
Worse still, a letter in defence of the programme that appeared in the Guardian, turned out to have been written by the BBC and was not even seen by some of its "signatories" -see .Alternative Medicine series: dirty tricks at the BBC? All this took a lot of work and got nowhere.
Simon Singh, the eminent science author, wrote two articles that exposed the very misleading portrayal of anaesthesia with acupuncture In the Guardian he wrote A groundbreaking experiment … or a sensationalised TV stunt?, and in the Daily Telegraph he wrote Did we really witness the ‘amazing power’ of acupuncture?. Singh also sent complaints to the BBC, but he persisted after the complaints were fobbed off and eventually his complaints reached the BBC Trustees. Two of his three serious complaints were upheld.
Phone-in programmes are notoriously bad for both balance and phony balance. In the alternative medicine field, equal time is always given to scientists, astrologers and crystal healers. The presenters are usually ill-informed and the callers are usually even less well informed. A particularly bad example follows.
Call You and Yours. The Radio 4 programme, You and Yours, at its best, can be quite good. It did a good job on a "snoring remedy" that I’d investigated, though it omitted some things that should, in my view, have been included. But they also have phone in versions of the programme. On 29th February 2010, they ran a phone-in programme about herbal medicine, hosted by Julian Worricker, someone who clearly was totally unaware of the controversies that surrounded this subject and, particularly, its regulation. The worst thing about this programme was that it featured a resident ‘expert’ That was Michael McIntyre who is chair of the European Herbal & Traditional Medicine Practitioners Association. McIntyre is a well known advocate of alternative medicine, who constantly fudges the need for proper evidence.
I went through the programme carefully, making detailed notes, which appear at Some truly appalling reporting of science by the BBC. It was one of the most biassed programmes on the topic I have ever heard. i sent a complaint to the BBC, referring to the detailed analysis which had already appeared here. To my dismay, they wouldn’t accept a complaint in the form. They wanted me to type the whole thing in a little box on the complaint site. where there is no formatting and no live links. I protested in vain that if they wanted a printed version, all the had to do was print the web page. At this point I decided that there was no point in spending yet more time to cope with the inflexibility of the complaints procedure.
Today programme. I’m an avid listener toToday, the best news programme on radio John Humphrys has no greater fan than I. For politics it is superb. But for science it is, sad to say, not always so good. One reason is that the presenters don’t know enough about the topics to ask the same sort of tough questions that they fire at politicians. Another reason is that they suffer badly from the phony balance problem (see below). A third reason is that they tend to pick up on silly survey press releases (the sort of ‘men with long big toes are better in bed’ pseudo-science); They may quite rightly laugh at them but this sort of thing doesn’t count as science reporting.. The Today programme is admirably serious about politics, but the science is often dumbed down and uncritical.
What needs to be done to improve BBC science
Link to the sources. Despite pressure from bloggers, the BBC web site still does not usually link to original sources, the paper on which claims are based. The whole virtue of the web is that it makes this very easy to do.
Anonymity of reporters. Too often reports of science on the BBC web site are anonymous. There is no excuse for that. Every report should carry the name and email address of the person who wrote it, Most newspapers do this, but the BBC is lagging behind.
Reaction to criticism. In most cases that I’ve tried, the reaction to constructive criticism has been obstructive and defensive. Producers seem very reluctant to admit that any mistakes were made. That needs to be changed.
Science correspondents are too often uncritical. A few more with the approach of investigative journalists would improve standards. An example is provided by a recent report “It’s good to think – but not too much, scientists say“. This is typical of the sort of work that many people find a bit hard to take seriously, but the report reads a bit like a regurgitation of press releases. There is no link to original sources and no attempt at evaluation.
Press releases. One reason for misleading reports stems from misleading press releases. Press releases often come from media departments who regard their job as getting their university into the headlinse, rather than explaining science. Worse still, sometimes the misleading hyps stems from the authors themselves (one example here, but there are hundreds to choose from). This makes it very important that science reporters should read the paper and have good enough critical faculties to read through the hype.
Complaints procedure needs to be improved. Complaints should be accepted in any form, The present web form is suitable only for short and simple criticicisms. An email address should be provided and it should accept attached documents. Certainly complaints in the form of web pages should be welcomed, because the live links provide the simplest way to refer to source documents.
The problem of phony balance. This is biggest problem of the lot.
In the wake of the report by the Science and Technology Committee (STC) on the lack of evidence for homeopathy, and the Chinese medicine poisoning, the BBC carried at least three very bad reports. Being a strong supporter of the BBC that saddens me. These cases are summarised at Some truly appalling reporting of science by the BBC. The worst was the case of Call You and Yours. There was also a totally imbalanced and ill-informed report on statutory regulation, and a very irresponsible video of a woman who claimed homeopathy cured her cancer. .
The question of balance is important. Ofcom imposes an obligation that reporting should reflect the balance of viewpoints. Section 5 of Ofcom’s broadcasting code says (emphasis is mine).
“Section 5: Due Impartiality and Due Accuracy and Undue Prominence of Views and Opinions”
“To ensure that news, in whatever form, is reported with due accuracy and presented with due impartiality.”
“Meaning of “due impartiality”: “Due” is an important qualification to the concept of impartiality. Impartiality itself means not favouring one side over another. “Due” means adequate or appropriate to the subject and nature of the programme. So “due impartiality” does not mean an equal division of time has to be given to every view, or that every argument and every facet of every argument has to be represented.”
The BBC Trust has a very similar definition of “due impartiality”.
Thus the rules stare quite explicitly that "impartial" does not mean giving equal time to any view, however batty,
In practice, though, producers often seem to play it too safe, and choose to give the same time time to the view that the earth is flat as is does to the view that the earth is spherical (OK, an oblate ellipsoid). This often gives a quite misleading impression of the state of play of informed opinion. Inappropriate use of “equal time” is the most common cause for misleading reports
Minority views should be heard of course, but they should not be given equal prominence to views that are held by the vast majority of informed people. Inevitably the worst cases arise in the areas of quack medicine, climate change and evolution.
Somebody said recently, it is as though after an air crash one gave equal time to the air accident investigator and a representative gravity-deniers association. That is scarcely an exaggeration of what happens on the BBC too often.
Worse still, far more time was given (especially on ‘Call You and Yours’) to the viewpoint that any scientist, indeed any informed person, would regard as quackery.
One thing that could be done about this false balance is to have better informed producers, or, more likely, to have better informed science reporters who can give advice on the state of opinion (and to make sure that their advice is sought).
Unless the BBC starts to be more critical in some of its reports, it could lose its preeminence. In the last few years it has become increasingly the case that the best critical evaluations of science are to be found not in the BBC or other mainstream media, but on blogs written by working scientists. Perhaps the BBC should ask them more often than it does at present.
Now give your opinions, below or email them to firstname.lastname@example.org
The Open University is a wonderful institution
I should have made it clear the Open University has played a big role in producing some of the best programmes. I was, quite rightly, corrected by a letter from an OU scientist. I’ll quote from it.
David Attenborough’s series on Life In Cold Blood, Charles Darwin and the Tree Of Life, Life – these are all Open University commissions. Life In Cold Blood also won a BAFTA and we have a string of other awards. I believe this is some of the very best science broadcasting the Open University puts out and I’m not surprised to see it at the top of your list.
More Or Less is an Open University commission – see e.g. http://www.open2.net/moreorless/
Material World is another one of our occasional commissions -see e.g. http://www.open2.net/materialworld/index.html
Geoff Watts contributed to the BBC Darwin season last year, which was heavily supported by Open University programmes.
See for example http://www.bbc.co.uk/darwin/
We support the series with a considerable amount of on-line material both for credit and not for credit. This material is accessible to the public at these open2.net sites, and at http://www.open.ac.uk/openlearn/.
It is clear that the criticisms levelled at the Alternative Medicine series are very much the exception to the usual excellent work of the OU. I’m told that that programme had nothing to do with the science faculty. Clearly it was an unusual aberation. I presume it was connected with the OU’s course K221, which I wrote about in 2006, under the title Open University Quacks. That sort of thing is quite atypical of the Open University, and something of an embarrassment to the many top rate people who work there. As usual, the blame lies not with scientists but with senior managers. After hearing about course K221, I had a long correspondence with Professor David Vincent, a pro-vice chancellor. He made sympathetic noises, but did absolutely nothing. That’s par for the course with senior administrators.
The Open University has been a magnificent success from the outset. Its first vice-chancellor was Walter Laing Macdonald Perry . Before he took that job, he was professor of Pharmacology in Edinburgh (and one of my Ph.D. supervisors). He did a great job.
Failure to report negative results
A classic example of a sin of omission by the BBC (and the rest of the mainstream media too) occurred recently in the reporting of the alleged effect of B vitamins on the development of Alzheimer’s disease. A positive trial was widely reported, but two weeks later a trial appeared that measured the eight thing -cognitive deficiency – and that showed no effect at all. As far as I can tell it was barely reported at all, The details are at http://www.dcscience.net/?p=3516
On July 1st. the journal Nature, published three reports that described genetic differences between people with schizophrenia and the rest of us. Nature held a big press conference, at the World Conference of Science Journalists, Many of the individual institutions involved in the studies also issued press releases. As so often in press releases, descriptions like “landmark”, “major step forward” and “real scientific breakthrough” were used liberally.
In short, there is no simple genetic basis for schizophrenia. The pooled results of the three studies gave 8,014 cases and 19,090 controls. A huge amount of genetic analysis was involved. Many thousands of small genetic differences were found between the schizophrenics and the controls, but no single mutation alone had much effect. There could be even more, because it is still not possible to sequence the entire genome of so many people. . In total, the effects of the many small differences might account for 30 percent of the disease risk. That’s more than the few percent that could be predicted before, though it’s still well short of the 70 or 80 percent figure that is often quoted for the genetic component of schizophrenia.
What is observed is a correlation between differences in the genome and having schizophrenia. The results don’t tell us directly about the cause of schizophrenia, and they don’t even (so far anyway) tell us much about the mechanisms that give rise to schizophrenia.
The reporting of this event makes a fascinating contrast in styles of scientific journalism. I’ll compare only two of the reports. One is by Steve Connor in the Independent (he of the recent contretemps). The other is by Nicholas Wade in the New York Times.
The headline sounds pretty hyped up to me, but headlines are often written by sub-editors, not by the author of the article. The article itself seems to be a pretty straight account that uses mostly quotations from the press releases. And it is accompanied by an irrelevant picture of a brain image, that’s alleged to show areas of the brain that contain dopamine receptors, though the work being discussed shows exactly that It’s not just a matter of a disorder in a transmitter system.. So the article is OK but it doesn’t contribute anything itself.
The fact of the matter is that the results of the gargantuan effort put into this study is to show how enormously complicated the problem is and just how little we understand, The prospect of inventing ‘cures’ looks almost hopeless in the face of such complexity. Nobody said it would be easy, but it has turned
out to be even harder than pessimists guessed. I would prefer a report that conveyed this reality better than the press releases do, and better than most of the hundreds of newspaper reports based on the press releases. There is one such report, that in the New York Times.
The journal Nature held a big press conference in London Wednesday, at the World Conference of Science Journalists, to unveil three large studies of the genetics of schizophrenia. Press releases from five American and European institutions celebrated the findings, one using epithets like “landmark,” “major step forward,” and “real scientific breakthrough.” It was the kind of hoopla you’d expect for an actual scientific advance.
It seems to me the reports represent more of a historic defeat, a Pearl Harbor of schizophrenia research.
The defeat points solely to the daunting nature of the adversary, not to any failing on the part of the researchers, who were using the most advanced tools available. Still, who is helped by dressing
The principal news from the three studies is that schizophrenia is caused by a very large number of errant genes, not a manageable and meaningful handful.
The rationale behind the long search for schizophrenia genes was entirely justifiable. Since schizophrenia is highly heritable, it must have a strong genetic component. And it has long seemed possible that the responsible genetic variants underlying most common diseases would also be common. Natural selection gives us strong protection against diseases that strike before the age of reproduction. But its power to eliminate harmful genes is thought to wane sharply thereafter. So bad versions of genes that are bad only late in life could build up in the population, explaining why the common diseases that strike later in life are so common.
And if researchers could identify the few major variants assumed to underlie each of these common diseases, from schizophrenia to heart disease to cancer, they could devise drugs to offset the genes’ effects.
But nature is often a lot more complex than assumed. It now seems that the arm of natural selection is far longer than thought. It has reached way beyond our reproductive years and zapped most harmful genetic variants before they could get to be common in the population. That leaves relatively uncommon variants, lots and lots of them in each case, as the genetic cause of each common disease.
In the last few years gene hunters in one common disease after another have turned up a few causative variant genes, after vast effort, but the variants generally account for a small percentage of the overall burden of illness. With most common diseases, it turns out, the disease is caused not by ten very common variant genes but by 10,000 relatively rare ones.
Today it’s the turn of schizophrenia researchers to make the same discovery, though one perhaps more to be expected since schizophrenia is not good for reproduction.
Schizophrenia too seems to be not a single disease, but the end point of 10,000 different disruptions to the delicate architecture of the human brain.
Yes, that discovery is a landmark. The kind that says you have 10,000 miles yet to go.
The march of science is not direct but two steps forward, one step back. This was the step back. But it was a completely necessary one. So the press release writers could have cast it as a noble defeat, were words like defeat a part of their vocabulary, or frankness their masters’ priority.
This account may sound a bit pessimistic, but it also seems to me to be the most realistic of the lot.
Schizophrenia is not alone in proving to be a lot more complicated than one would have hoped. One of the most interesting outcomes of the genomic age has been to show that far simpler conditions than schizophrenia turn out to be not so simple after all. The simplest genetically transmitted diseases are those caused by mutation of a single amino acid in a single protein of known function. For example, one of the best understood receptors is the type of acetylcholine receptor that is responsible for transmitting an impulse from a motor nerve to a muscle cell, Curare blocks them and so paralyses voluntary movement, A rare form of muscle weakness, slow channel congenital myasthenic syndrome (SCCMS), is caused when one of the amino acids in the protein mutates. But it has turned out that there is not just one mutation. It would be nearer to the truth to say that each family that suffers from the disease has its own mutation. Each of the mutations has a rather similar effect on the function of the protein, but there is not just one SCCMS but dozens.
The same is true of mutations in the glycine receptor that cause the rare congenital condition, ‘startle disease’ (posh name, hyperekplexia), The glycine receptor mediates inhibitory actions in the spinal cord. It is blocked by strychine, Strychnine causes exaggerated reflexes and eventually tonic convulsions: your muscles tighten so you die with the risus sardonicus. Mutations in the glycine receptor that stop it working so well have an effect rather like strychnine. But rare though the disease is, at least 15 different mutations, each in a different family, have been found to produce similar effects. There is just one startle disease, but many.
Cystic fibrosis is even more complicated. Mutations in a single protein, with a function that is now quite well understood, can cause the disease. But every patient does not have the same mutation. Around 1500 mutations have been found in the 20 years since the gene was sequenced. And to make it still more complicated, the symptoms shown by two patients with the same mutation may not be identical. The effects depend, it seems, on the rest of the genetic make-up of the individual. There is a nice account in ‘The promise of a cure: 20 years and counting’..
The same pattern is repeated again and again. Schizophrenia is just a very extreme example of a common phenomenon Not so long ago it was the dogma was that the future of drug discovery lay in genomics and high-throughput screening, Richard Sykes, when head of GSK, put the vast resources of GSK into this approach. It did not really work (which might explain why he left GSK to become rector of Imperial College). Now we know why.
It turns out to be a bit more complicated than anyone had foreseen, and one suspects it will take a long time to sort it out.
There is some comment on the reports on the Nature web site.
As so often, there is a first class level-headed account of the facts and what they mean on the NHS Choices site.
Acupuncture in the BMJ
- The analgesic effect of acupuncture is small and cannot be distinguished from bias resulting from incomplete blinding.
- The analgesic effect of placebo acupuncture is moderate but very variable as some large trials report substantial effects.
- The effect of acupuncture seems to be unrelated to the type of placebo acupuncture used as control.
The results confirm, yet again, that there is essentially no difference between “real” acupuncture and sham acupuncture. All that talk about meridians and Qi really is so much mumbo jumbo.
The average effect (the diamond at the bottom) is essentially zero.
It has often been supposed that acupuncture is a theatrical placebo, but because of the placebo effect it produces more pain relief than (non-blind) controls. This study confirms that there is likely to be such an effect, but it also finds that the size of the placebo effect is too small to be useful to patients. Here is the comparison between sham acupuncture and no acupuncture at all.
The results of different trials are very variable, but the average effect (diamond at the bottom) favours sham acupuncture over no acupuncture at all.
But how big is the effect? The numbers along the bottom of the graph are ‘standardised mean differences’. The average value of -0.42 (95% confidence limits -0.6 to -0.23) between ‘no acupuncture’ and ‘sham acupuncture’ corresponds to a difference of about 10 points on a 100 point scale. This difference is big enough to be real, in the sense that it isn’t just chance. But is it big enough to be useful to the patient? Probably not. Madsen et al conclude
“a consensus report characterised a 10mm reduction on a 100 mm visual analogue scale as representing a “minimal” change or “little change”. Thus, the apparent analgesic effect of acupuncture seems to be below a clinically relevant pain improvement.”
This makes nonsense of the Pittilo report. Notice that these results, yet again, make nonsense of the proposals in the gamma-minus Pittilo report, to pseudo-regulate acupuncture, and to have degrees in the subject.
Cochrane reviews of Acupuncture : and a bad report from BBC
You can’t blame subeditors for the appalling title of the BBC’s report on Acupuncture ‘works for headaches’. The content is pretty misleading too. (The link is to a version saved at 19.35 on 21 January 2009.). Furthermore, like far too many of the BBC reports, it is anonymous. One has no idea to blame This is important, if only because the BBC news site is so influential. Twelve hours after posting the misleading title has been copied all round the world.
In the Guardian, Ian Sample had a much better report, Even’fake’ acupuncture reduces the severity of headaches and migraines. Sadly the print edition had the title “Acupuncture aids migraines, researchers
find”, but that can be blamed on subeditors who have a problem with reading (in fact Sample had seen neither title).
The BBC report seemed to call for a complaint. This is what I sent.
|I wish to complain about the report on acupuncture at http://news.bbc.co.uk/1/hi/health/7838231.stm
The title itself was highly misleading “Acupuncture works for headaches” is precisely the opposite of what was shown, namely that it is no better than sham acupuncture controls.
The article goes on to say
It is not said that there isn’t the slightest reason to think this is true (or even means anything) , and that the work which is being reported is strong evidence that it’s not true
That is wrong. University of Salford , for example has just closed its acupuncture course after a number of us have pointed out that it is teaching things that aren’t true
It is exactly the opposite of an endorsement. It is one more nail in the coffin of acupuncture as generally understood.
This article, I contend, is partial and misleading. Unfortunately it is anonymous.
Today’s BBC report on the BMJ paper is almost as partisan as the one that I just complained about. “Confusion on acupuncture benefit ” exaggerates greatly the amount of confusion. Worse still, it quotes only two well-known advocates of acupuncture, people who make their living from it. No independent scientific voice is to be heard.
One of those cited is Adrian White. He is described as “a researcher into acupuncture at the Peninsula Medical School”. There is no mention of the fact that he is also Editor in Chief of the journal Acupuncture in Medicine. The other person who is quoted is Mike O’Farrell, the chief executive of the British Acupuncture Council. What do you expect him to say?
I’ve just sent another complaint, about today’s BBC report. The more the merrier.
The BMJ itself published an editorial on the Madsen paper. It seems very odd that they should have chosen the editors of the BMJ-group journal, Acupuncture in Medicine, to comment on a paper that sounds the death knell for the subject from which they make their living. Needless to say, the editorial attempted to wriggle out of the obvious conclusions. Worse still, the Health editor of the BBC web site referred to the editorial in his defence, in response to my complaint.
Channel 4 News does a lot better than the BBC, or the BMJ editorial, with its report Acupuncture ‘fails to relieve pain’
“Claims that acupuncture can relieve pain have been undermined by the results of a new study.”
“senior researcher Asbjorn Hrobjartsson said: “Our findings question both the traditional foundation of acupuncture…and the prevailing theory that acupuncture has an important effect on pain in general”. “
The Daily Telegraph also, like almost everyone else, did better than the BBC, with “Acupuncture ‘has almost no effect in relieving pain’”
“The pain relieving effects of acupuncture are so small that they may be clinically irrelevant, according to a review of research into the treatment.”
Even Metro, the free London Newspaper, is more accurate then the BBC. They carry a short report “Acupuncture ‘has no medical point’.”
“The pain-relieving effects of acupuncture compared with a placebo are so small they may be clinically irrelevant.”
New review puts in doubt traditional foundation of acupuncture. Believe it or not, that is the title of the report on Madsen et al from the Prince’s Foundation for Integretad Health! Don’t get too excited though. They haven’t even bothered to look at the original paper, but merely cite the dreadful BBC report. Much prominence is given to the acupuncturists, White and O’Farrell, and the important finding isn’t mentioned at all. Another gamma minus.
|One of the most extraordinary bits of journalism I’ve read for a long time appeared as an editorial in the Sri Lankan newspaper, the Sunday Leader, on Sunday January 11th 2009 It was reproduced in the Guardian on 13th January, and in The Times. It was written by Lasantha Wickrematunge, editor of the Sunday Leader, and it was the last thing he wrote. Days after writing it he was assassinated.|
It is a plea for freedom of speech. In particular, for the freedom of journalists to tell the truth, It is deeply moving and it is also written in more beautiful English than many native speakers can manage. The second person to leave a comment in the Guardian said
“Extraordinary, humbling and deeply moving.
Cif Eds, please leave this at the top of the page for about a week, and then nail copies it to every available surface at Guardian HQ.”
Writing blogs like this one (and a thousand others) need some of the skills of investigative journalism. Those skills are not so different from those you need in science, Curiosity, a willingness to look under stones, a preference for truth over myth, some skill with Google and a good deal of tenacity. You also need to be resilient to abuse and defamation by people who disagree with you. But you do not risk your life. It does not take much courage. That isn’t true in large parts of the world.
Read it all. Here are a few quotations to persuade you it’s worth the time.
“No other profession calls on its practitioners to lay down their lives for their art save the armed forces – and, in Sri
Lanka , journalism. In the course of the last few years, the independent media have increasingly come under attack. Electronic and print institutions have been burned, bombed, sealed and coerced. Countless journalists have been harassed, threatened and killed. It has been my honour to belong to all those categories, and now especially the last.”
“The Sunday Leader has been a controversial newspaper because we say it like we see it: whether it be a spade, a thief or a murderer, we call it by that name. We do not hide behind euphemism. The investigative articles we print
are supported by documentary evidence thanks to the public-spiritedness of citizens who at great risk to themselves pass on this material to us. We have exposed scandal after scandal, and never once in these 15 years has anyone proved us wrong or successfully prosecuted us.”
“The free media serve as a mirror in which the public can see itself sans mascara and styling gel. From us you learn the state of your nation, and especially its management by the people you elected to give your children a better future.”
“It is well known that I was on two occasions brutally assaulted, while on another my house was sprayed with machine-gun fire. Despite the government’s sanctimonious assurances, there was never a serious police inquiry into the perpetrators of these attacks, and the attackers were never apprehended.
In all these cases, I have reason to believe the attacks were inspired by the government. When finally I am killed, it will be the government that kills me.”
“In the wake of my death I know you will make all the usual sanctimonious noises and call upon the police to hold a swift and thorough inquiry.
But like all the inquiries you have ordered in the past, nothing will come of this one, too. For truth be told, we both know who will be behind my death, but dare not call his name. Not just my life but yours too depends on it.
As for me, I have the satisfaction of knowing that I walked tall and bowed to no man. And I have not travelled this journey alone. Fellow journalists in other branches of the media walked with me: most are now dead, imprisoned
without trial or exiled in far-off lands.”
“People often ask me why I take such risks and tell me it is a matter of time before I am bumped off. Of course I know that: it is inevitable. But if we do not speak out now, there will be no one left to speak for those who cannot,
whether they be ethnic minorities, the disadvantaged or the persecuted. An example that has inspired me throughout my career in journalism has been that of the German theologian, Martin Niemöller. In his youth he was an antisemite and an admirer of Hitler. As nazism took hold of Germany, however, he saw nazism for what it was. It was not just the Jews Hitler sought to extirpate, it was just about anyone with an alternate point of view. Niemöller spoke out, and for his trouble was incarcerated in the Sachsenhausen and Dachau concentration camps from 1937 to 1945, and very nearly executed. While incarcerated, he wrote a poem that, from the first time I read it in my teenage years, stuck hauntingly in my mind:
First they came for the Jews
and I did not speak out because I was not a Jew.
Then they came for the Communists
and I did not speak out because I was not a Communist.
Then they came for the trade unionists
and I did not speak out because I was not a trade unionist.
Then they came for me and there was no one left to speak out for me.
If you remember nothing else, let it be this: the Leader is there for you, be you Sinhalese, Tamil, Muslim, low-caste, homosexual, dissident or disabled.”
This man puts to shame the those who won’t speak out in the safety of the West, despite the fact that they have nothing to lose but their ministerial jobs or their knighthoods. Or running the risk of being sued by chiropractors.
How about some nominations for Western journalists who live up to these ideals? I’d start with Seymour Hersh and Paul Krugman in the USA, and our own Ben Goldacre. It’s interesting though, that two of these three are not full time journalists. Blogs do rather better than most newspapers. They have become an important force for freedom of speech. That more than counterbalances the use of the web for promoting junk. It is a lot harder to keep a secret than it used to be.
In a wonderful demonstration of common sense, the BBC has removed all the alternative medicine pages from BBC Health web site. I expect that it was helped in making that decision by the many complaints it had received about statements on these pages that were simply not true, The existence of these pages was just not compatible with the BBC’s commitment to accuracy.
|Needless to say, this decision was greeted with howls of rage from the alternative world. Some wrote to the BBC to complain (and I wrote to congratulate them). Until today I haven’t been able to find any BBC statement on the matter. This one appeared on Healthypages, one of the zaniest sites on the net. Nothing is too barmy for them. This is the picture they used to show how wrong the BBC is||
Picture from Healthypages
“I do a range of spiritual healing practices and can offer an energetic healing session including techniques from the Order of Melchizedek and about 10 forms of Reiki in exchange for a Theta Healing session.”
And just in case you don’t know about the Melchidezek method, here is a picture of a poster in the window of my local “health food” shop.
The explanation says
“Using ancient holographic technology, the basis of the techniques presented is the activation of the Merkaba, a rotating lightfield awakening your spherical consciousness. This raises one’s quotient of light vibration within the human atomic cell structure. Once activated. the merkaba assists us in accessing our naturally ascended consciousness state: the healing capabilities are enhanced a hundred-fold. The Hologram of Love has the ability to heal and rejuvenate any form of creation as it is the living conscious holographic pattern of God Source vibration.”
There, and I’ll bet you thought holography was a recent invention. As an example of sciencey-sounding words used in random order, this one takes a bit of beating. It really is an insult to human intellect.
Anyway, back to the BBC. This is their diplomatic response (what they should have said really was, ‘those pages were nonsense so we removed them’).
BBC issued the following statement, dated 22 Feb 2008:
We received complaints about the Complementary section of the BBC Health website being disbanded
The BBC’s response
The decision to remove the complementary medicine area of the health website was taken as part of a wider review of all the health content in order to enable the BBC to focus its efforts on creating new and exciting content.
In order to release resources for this redevelopment work, we’re reviewing existing content from an editorial and value-for-money perspective.The complementary health section was incomplete and, therefore, not of a satisfactory editorial standard.
It also represented a small proportion of traffic to the site but was disproportionately time-consuming.Therefore, the decision to take it down was based on a combination of factors: how much work it needed to maintain to a high editorial standard, how much this cost and how popular it was with site users.We have already removed other sections of the health site and plan to reduce or remove others.
We appreciate people are disappointed this area of the site has been removed and apologise if the decision has appeared abrupt to site users or inconvenienced other sites linking to BBC Health.
The BBC will continue to cover complementary health in other areas of its output, such as TV, radio and news programmes, and may reassess its complementary health content in future.
” . . . However the site has in recent months been targetted by the self-appointed ‘quackbusters’ ( . . . such as David Colquhoun), who sent a deluge of letters and emails claiming that complementary therapies such as homeopathy and cranial osteopathy were ‘unscientific’ and should be removed.”
Well thanks for the credit, but sorry, there was no deluge. I wrote no more than a couple of times myself, and I suspect that a handful of friends did the same. I didn’t even ask them to remove the whole lot, merely to correct particular statements that were not true,
‘Mardi’ goes on
“Rather than taking a reasoned view and considering the evidence from good research studies on complementary medicine, these individuals seem simply hell bent on trying to stamp out complementary medicine”
That really is a bit rich. Suddenly the alternative industry are invoking evidence from good research studies. It is precisely the other way round. It is because that evidence is almost all negative that the BBC have decided to remove their coverage.
Of course it may have helped that the BBC had to spend a lot of time defending itself against criticism of the BBC 2 TV series on Alternative Medicine from February 2006. After initially rejecting complaints, an appeal to the highest level, the Board of Trustees, two of the most serious complaints were upheld against these programmes.
One reason for bad science reporting is that journalists rely too much on press releases from university ‘media departments’. Their output often seems more akin to advertising than to science.
A recent example came from a poor report in the Independent that seemed designed to fuel “electrosmog” hysteria. But the press release from Imperial College was almost as bad. The details are on IMPROBABLE SCIENCE blog.
The Daily Telegraph recently published two reports about acupuncture. One said it worked. The other said it didn’t work. Needle cure effect ‘is not all in the mind’ By Catriona Davies, starts
Acupuncture has a genuine ability to relieve pain, scientists have found.
The other report was
Doubt cast on needle therapy for migraine By Nic Fleming, Health Correspondent, starts
Acupuncture is no better at reducing migraines than fake treatment, researchers say today.
A study involving more than 300 patients found the healing method did reduce headaches, but only by the same amount as placing needles at non-acupuncture points.
Unfortunately the Daily Telegraph‘s reporters did nothing to help the confused reader. No comment was made on the apparent contradiction. In this particular case, there is an obvious explanation.
The first (favourable) article was said to be published in Nature, though in fact it was published in Neuroimage (Pariente J., White P., Frackowiak , Richard S. J. & Lewith G. Neuroimage, 25. 1161 – 1167, 2005). Presumably the reporter had picked it up from a rather uncritical synopsis on the news site, nature.com. It was conducted on 14 patients with painful osteoarthritis. Contrary to the first line of the Telegraph‘s report (“Acupuncture has a genuine ability to relieve pain”), the work did not measure pain at all. In fact the summary of the paper says
“The three interventions, all of which were sub-optimal acupuncture treatment, did not modify the patient’s pain.”
What the paper actually did was to use positron emission tomography (PET) to measure ‘activation’ of certain parts of the brain when needles were inserted. Some parts were activated more by having acupuncture needles piercing ths body than by ‘stage-dagger’ needles which retracted and did not pierce the body. I have no criticism of these findings: my purpose here is to explore the apparent contradiction between this trial and another.
The second, unfavourable, report was of a much bigger study, 302 patients with migraine headaches. It was published by Linde et al. (Journal of the American Medical Association. 2005 293(17):2118-25). This study concluded
“Acupuncture was no more effective than sham acupuncture in reducing migraine headaches although both interventions were more effective than a waiting list control. ”
These two studies were on quite different conditions, used different methods, and very different numbers of patients. But suppose we take them at face value, are they not contradictory? No, not necessarily, because they used quite different sorts of control group.
The study that was reported as showing that acupuncture worked compared patients that had real acupuncture with patients who had treatment with “stage dagger” needles that appeared to pierce the skin but did not.
The study that was reported as showing that acupuncture did not work (Linde et al.) used a different sort of control group, acupuncture needles that pierced the skin but were inserted in the wrong places (as defined by the ‘principles’ of acupuncturists). No difference was found between ‘real acupuncture’ and control.
There is nothing incompatible about these two studies if one adopts the view that piercing the skin with a needle can produce a physiological response that makes the patient feel that other sorts of pain are less painful, but that it does not matter where the skin is pierced. The latter hypothesis means, of course, that all talk about “meridians”` and “energy flow” that acupuncturists use, is no more than mumbo jumbo.
Dr Lewith is quoted as saying, of the negative study, “This is a badly conceived study that just adds more confusion to the debate because it uses non-site specific acupuncture as a control.” Quite on the contrary, the Linde study seems more interesting to me, because it unveils the mumbo jumbo of meridians (at least for the particular points used in this study). Dr Lewith may find this conclusion unpalatable, but it is the obvious implication of this pair of studies.
Another study confirms that the principles of acupuncture are nonsense
A trial by Melchart and colleagues on 270 patients shows conclusively that acupuncture can indeed produce amelioration of tension-type headache when compared with no treatment, BUT the relief is produced whether or not needles are inserted at ‘acupuncture points’. Very similar results were found with ‘superficial needling at non-acupuncture points’. British Medical Journal, 15 August 2005. Yet again it is shown that the mumbo-jumbo of meridians and magic points is nonsense. Yet again, that is a good reason why universities cannot be expected to train acupuncturists. Our business is to minimise mumbo-jumbo, not to propagate it.