More or Less is an award-winning investigative radio show and podcast from the BBC. Presented by Tim Harford, the programme examines the accuracy of statistics in the news and popular culture. Their work is frequently praised by skeptics for its diligent deconstruction and debunking of misinformation. All of which made it more disappointing when, for their January 26th 2022 episode, More or Less seemed to get it so wrong in a brief conversation about the Placebo Effect.
The powerless placebo?
The segment opened with a guest recounting the story of Henry Beecher, a US Army doctor who, we are told, ran into a shortage of morphine while working as a surgeon during World War Two. Forced to administer inert salt water injections instead, his patients nevertheless experienced – says More or Less – ‘an astonishing amount of pain relief.’
Unfortunately, there is little evidence this story is true. When the journalist Shannon Harvey set out to verify the tale, she was unable to find it anywhere in the corpus of Beecher’s published work, or in his personal archive held by Harvard. Rather the story appears to be a corruption of a case study where Beecher administered a sedative to a wounded man instead of a painkiller, reasoning his anxiety was contributing more to the pain than the wound itself. There was no morphine shortage, or inert injections.
That aside, Beecher is responsible for probably the single most influential paper in placebo effect research. Published by JAMA in 1955, The Powerful Placebo cites ‘fifteen illustrative studies’ involving 1,082 patients, which found that placebos have an average significant effectiveness of 35.2 ±2.2%, ‘a degree not widely recognised.’
In 1997, the Journal of Clinical Epidemiology published a modern re-analysis of Beecher’s fifteen papers. The authors, Kienle and Kiene, identify a raft of things which could give a false impression of a placebo effect. For example there is spontaneous improvement; some patients were just going to get better anyway. As Kienle and Kiene say:
Spontaneous improvement of a disease does not occur as a result of a placebo administration; it is not an effect of a placebo. This often seems to be disregarded in placebo literature.
One of the papers cited by Beecher is Medicinal Treatment of the Common Cold, published in 1933. Beecher looks at the patients in that study whose colds improved after taking placebo and ascribes that recovery to the placebo effect, completely failing to account for the fact that colds get better after a few days anyway. According to Kienle and Kiene, ten of the fifteen studies cited by Beecher fail to account for spontaneous improvement.
Then there is regression to the mean. Some of the improvement observed will be the result of the fluctuation of symptoms, yet Beecher ascribes to the placebo effect improvements seen in five of his fifteen papers without accounting for regression.
Parallel interventions are where patients take something else, some other active treatment, at the same time as the placebo. Relatedly, unrecorded parallel interventions are where patients not only take something else, but they don’t even tell their doctor they’re doing it. Or if they do, the doctor doesn’t write it down. In two of Beecher’s fifteen studies, patients were receiving other active interventions alongside the placebo, so the effects observed cannot simply be ascribed to the placebo effect alone.
Finally there is misquotation. Kienle and Kiene claim Beecher misquotes ten of the fifteen papers:
He sometimes inflated the percentage or the number of patients, or he cites as a percentage of patients what, in the original publications, is referred to as something completely different, such as the number of pills given, the percentage of days treated, the amount of gas applied in an experimental setting or the frequency of coughs after irritating a patient.
In fact none of the fifteen papers cited by Beecher demonstrate a convincing effect for placebo which could not be fully explained by some other means.
The increasing power of placebo
After a short sidebar on how placebos can change our blood pressure, or prompt the release of endorphins – claims to which I have no objection, since these physiological effects can plausibly be brought on by purely psychological interventions – the conversation then moves on to how the placebo is ‘getting more powerful.’
More or Less cites a review which looked at the effects of painkillers across 80 studies between 1990 and 2013. From More or Less:
… at the start of those trials, the real painkillers outperformed the placebo by about 27%. By 2013, they only outperformed the placebo by 9%. And what was astonishing there was that the effects of the drugs themselves had remained pretty steady.
This was tentatively attributed to direct advertising influencing the expectations of patients. People expect more from their drugs, but they also expect more from their placebos.
Perhaps a more parsimonious explanation for this ‘decline effect’ is that we were simply better at designing and conducting rigorous clinical research in 2013 than we were in the early 90s. A poorly designed study, which fails to capture some non-specific effects in the control group, would appear to depress the effect seen in the placebo arm and therefore exaggerate the effect size in the treatment arm. It’s not that placebos are getting more powerful, it’s that we’re getting better at using them to control our trials.
Mind-set matters
Finally, the conversation turned to a 2007 study which followed 84 women working as hotel cleaners. The paper, by Crum and Langer, was titled ‘Mind-set matters: exercise and the placebo effect’ and was published in the journal Psychological Science.
Each of the 84 women worked between 32 to 40 hours per week, and each cleaned approximately fifteen rooms per day. Crum and Langer took half of the women and gave them a presentation which said the cleaning they did each day is good exercise and actually meets the Surgeon General’s recommendations for an active lifestyle. The other half were not given this information and were left to keep cleaning hotel rooms as they were.
By the end of the study, the group who were told cleaning was good exercise had significantly lower mean weight, body mass index, waist-to-hip ratio, body fat percentage and blood pressure. They also reported that they were doing more exercise than before.
Except that the participants did not report they were going to the gym more often, or walking to work more. They reported getting more exercise, but did not report any obvious change in their activity. And when the authors asked the housekeeping managers if their staff were cleaning any harder than before, they were told that they weren’t. Crum and Langer concluded:
Therefore, if there was no increase in reported exercise outside of work, any increase in perceived regular exercise, perceived amount of exercise, or perceived work as exercise would be assumed to reflect a change in mind-set initiated by the intervention and not due to an increase in actual exercise.
But do we really know that the participants were not doing any more exercise? Perhaps they were cleaning rooms with a little more gusto, chipper in the knowledge that it was actually doing them some good? Perhaps they had a more positive approach to their work, so they cut fewer corners when cleaning a room?
These kinds of subtle effects and biases were very poorly controlled for – essentially the control was asking the boss if the women worked harder. But would they really know? They don’t follow the workers room-to-room to watch how hard they’re scrubbing. This is also a study with small numbers, mostly based on self-reported data. What we really need to see, before we conclude that your mind will make you lose weight, is a replication of this study.
Fortunately for us, a few years later someone did exactly that. An investigation of exercise and the placebo effect was published in the American Journal of Health Behavior in 2011, as a replication of Crum and Langer. Fifty-three university campus building service workers were recruited and split into two groups. As before, half of them were told that their work counted as good exercise and half were not. After eight weeks, the authors found no difference between the groups. From that paper:
Groups demonstrated similar outcomes at 4 and 8 weeks for weight, percent fat, waist circumference, and behavioural measures. This research did not support the placebo effect. Although enticing, simply changing mind-set does not alter the relationship between exercise and health.
Far from telling us ‘about how our interpretations of all the things we do in life could be really important for our health and well-being,’ as claimed by More or Less, what this tells us is to be wary of apparently fantastical but spurious results in small and badly controlled papers.
