Skip to content

Anecdotes as Evidence

March 24, 2010

I had an interesting chat last night. I can’t remember how it came up, but acupuncture was mentioned. I may have slightly scoffed at this. Personally, I categorise it in what Mark Crislip at Science-Based Medicine calls Supplementary, Complimentary and Alternative Medicines (SCAMs). Why do I dismiss it? It has no plausible mechanism (it relies on a theory involving “qi” or “vital energy”, not that anyone could point it out to you). Studies have repeatedly shown that acupuncture is no more effective than an elaborate sham acupuncture that acts purely as placebo (see for instance here and here). So whilst it may work in the sense that it makes some people feel better, it is not efficacious in terms that would make it medically worth while.

So, I made it clear that I think acupuncture doesn’t work. But, said my friend, you’re ignoring my evidence! It’s worked for me, and here’s a story of when it cured me of condition x…

Ah, I said, here’s a story of when it didn’t cure my sister of condition y. So, it was one-all on the anecdote front. But despite this, I was still accused of being irrational for dismissing evidence. This raises an interesting point that I don’t think many “laypeople” understand, which is that there many levels of evidence in medicine, some of them worth more than others.

At the bottom of the pile is anecdotal evidence. You can always find a story of someone who thinks that a treatment has worked for them. Homeopathy, chiropractic, acupuncture, all supported by anecdotal evidence. But humans are easily fooled, and any one story isn’t an objective way of analysing efficaciousness.

At the top of the pile is the randomized, controlled, double-blind trial. A good discussion of what this involves is here. There are still problems. Studies vary massively in design quality and it’s pretty hard for you or I to decide what is good and what is bad. Sometimes results are generated by chance that can go against the grain.

These difficulties are partially overcome by meta-analysis. By combining the results of the highest quality studies (in a clear, transparent way) we can get a clear view of exactly how effective a treatment is. This is the way in which, for instance, homeopathy can be shown to be less and less effective the better designed your study is.

I tried explaining this, but to no avail. Ignoring someone’s story was viewed as worse than accepting bad evidence. This is a major problem not just in medicine but in many other aspects of life. By following the scientific method we take away the personal touch that SCAMs, religion and other pseudo-science can give people. The fact that I can provide an alternative “story” doesn’t matter if it’s not viewed as relevant to you. This is going to be a major stumbling block for science until people realise they’re not special, not here with a god-given purpose and that solipsism can be taken too far.

2 Comments leave one →
  1. Nicky permalink
    March 29, 2010 9:44 PM

    I’ve had a similar discussion with a friend of mine who went to a nutritionist (or homeopath or one of those quack types) who after chatting to him for a hour (and charging him god (or other higher being of choice) knows what) told him he was dairy intolerant. No tests. No check up. Nothing. Yet he took this as truth & now doesn’t eat dairy products. Apparently he feels better for it but can’t really explain what better really means.

    But this brings up an interesting problem which I have recently stumbled across in my own existence. A long-held belief of mine about vitamin C tablets & colds was crushed after reading Bad Science. I casually mentioned to Carl the other day that peppermint tea helps digestion. He asked me how I knew this. I said I didn’t know I just did! Having only recently discovered the idea of a randomized, controlled, double-blind trial (I probably learned it at school but I’m getting on a bit now & that was a while ago) it has blown my small mind how much I take for granted & how wrong I probably am about so much. How do I know I can even trust Dr Goldacre?! Short of reading published trials on everything I believe to be true or conducting my own experiments, how can I believe anything? There’s just too much stuff out there!

    So even though I think my friend is completely dumb for heeding the advice given to him, I understand that wading through all the crap to get to the correct stuff is hard.

  2. March 30, 2010 7:42 AM

    I agree, it’s largely only by accident that I’ve discovered that things I thought were good aren’t. Searching Science-Based Medicine turns up answers to a lot of common questions. There’s also a pretty awesome interactive graphic at Information is Beautiful that ranks the evidence for different “cures”.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: