Acupuncture appears to show results.

[politas]

One way to handle it could be to use four groups:

Group 1: Receives acupuncture; are told they are receiving acupuncture
Group 2: Receives needling; are told they are receiving acupuncture
Group 3: Receives acupuncture; are told they are receiving fake acupuncture
Group 4: Receives needling; are told they are receiving fake acupuncture

Then compare the differences between groups 1 & 3 to the differences between groups 1 & 2 and the differences between groups 2 & 4.

If the people in group 3 report significantly less improvement than group 1, while groups 2 and 1 report similar improvement, that would be pretty devastating, especially if the difference between groups 2 & 4 are close to 1 & 3.

You'd pretty much have to do it by telling the patients whether they received acupuncture for their complaint or a different complaint after the treatment, and have the acupuncturists all speaking a different language to the patients, so the acupuncturists are told which symptoms they are to treat without the patients able to hear. I don't know how to control for non-verbal communication, though.

 

[BillyJoe]

One way to handle it could be to use four groups:

Group 1: Receives acupuncture; are told they are receiving acupuncture
Group 2: Receives needling; are told they are receiving acupuncture
Group 3: Receives acupuncture; are told they are receiving fake acupuncture
Group 4: Receives needling; are told they are receiving fake acupuncture

The first problem is that two of your groups (groups 3 & 4) are not even testing for acupuncture, but for the placebo effect.
The second problem is that you are going to halve the number in each of the two groups that are actually testing for acupuncture thereby significantly decreasing the strength of your trial of acupuncture.

Then compare the differences between groups 1 & 3...

This is a test of the placebo effect and tells you nothing about the effect of acupuncture.

....to the differences between groups 1 & 2

This is the proper test for the effect of acupuncture.

and the differences between groups 2 & 4.

Again, this is a test for the placebo effect.

If the people in group 3 report significantly less improvement than group 1, while groups 2 and 1 report similar improvement, that would be pretty devastating, especially if the difference between groups 2 & 4 are close to 1 & 3.

Comparing the differences between these groups is unlikely to show clearly what you want it to show, especially when you have significantly reduced the strength of your trial by halving the numbers in each group. Typically about 100 patients are recruited in such trials. If you stick with groups 1 & 2 (the two groups that will actually test for the effect of acupuncture), you will have 50 in each group which is considered a minimum requirement.
Having more groups, and less in each group, usually ends up obscuring any useful results and is entirely unnecessary.

You'd pretty much have to do it by telling the patients whether they received acupuncture for their complaint or a different complaint after the treatment...

For ethical reasons, patients must be told they will receive either true or sham acupuncture before the trial actually commences.

...and have the acupuncturists all speaking a different language to the patients, so the acupuncturists are told which symptoms they are to treat without the patients able to hear.

I agree that the main difficulty is blinding the acupuncturists, but the trick is to ensure that they do not know which is the true acpuncture and which is the sham acupuncture or, in other words, which condition these patient are actually suffering from. The only way is to prevent verbal communication between the acupuncturists and the patients about what disease/symptom they are suffering from.

I don't know how to control for non-verbal communication, though.

If you can prevent verbal communication, you have effectively prevented non-verbal communication.

 

[Sherman Bay]

I've written several posts on this before, but I think part of the problem is that there isn't really a placebo effect, per se. The improvements that we see in a placebo group are due to a hodge podge of different things, the bulk of which is simply regression to the mean and bias.

Bausell spends quite a bit of time on exactly those biases. But if you deny that the placebo effect exists, you are denying his major premise and the bulk of his book.

One of the things I found refreshing was his admission that the placebo effect does exist, whether mind over matter or just mind over mind. With that out of the way, we can concentrate on other things. He shows illustrations, via charts, some hypothetical, on ways to subtract placebo "offsets" from test results to obtain less-biased figures.

However, as you suggest, the placebo effect does NOT exist, I don't feel it invalidates Bausell's arguments on data accuracy.

Are you now saying that Bausell, Carroll and hundreds of thousands of controlled clinical trials are wrong?

Are you saying that everyone is right without individual examination or question?

 

[fls]

Bausell spends quite a bit of time on exactly those biases. But if you deny that the placebo effect exists, you are denying his major premise and the bulk of his book.

I didn't mean to (hence my caveat at the end of my post). The major premise of his book is that the improvements seen with complementary therapies are the same as the improvements seen with placebo. And as you pointed out, he talks about the various reasons for those improvements. He does spend two chapters talking about whether there is also an effect specific to placebo (i.e. "is there really such a thing as an analgesic placebo effect?"). And I think that it is reasonable to consider that a 'placebo effect'. My point was that most of the improvements seen in the placebo group are due to something other than that specific placebo effect, and that in many studies it isn't even part of the improvement that is measured.

One of the things I found refreshing was his admission that the placebo effect does exist, whether mind over matter or just mind over mind. With that out of the way, we can concentrate on other things. He shows illustrations, via charts, some hypothetical, on ways to subtract placebo "offsets" from test results to obtain less-biased figures.

However, as you suggest, the placebo effect does NOT exist, I don't feel it invalidates Bausell's arguments on data accuracy.

Bausell and I are really saying the same thing. I think he does a good job of showing how you begin to separate out all the different things that lead to improvement and how you can address the issue of whether there is an effect specific to placebo. And you are right that none of his arguments depend upon whether or not there really is a specific placebo effect.

Are you saying that everyone is right without individual examination or question?

I'm saying that two people you seemed to be willing to consider authoritative on this issue don't think an untreated control group is necessary in order to draw conclusions about treatment effectiveness. And that most clinical trials do not include an untreated control group, yet physicians still feel comfortable drawing conclusions about treatment effectiveness. If you think that an untreated control group is necessary, I think you need to make some attempt to address why so many people with knowledge and experience on this issue have it wrong.

Linda

 

[Skeptic Ginger]

...
All treatments have a placebo effect, ....

I take issue with this claim. There is some evidence paying attention to things when doing research affects some outcomes, and there is evidence of a placebo effect on subjective assessments especially pain. But there is not good evidence that all treatments have a placebo effect. For example, you would want to see some good evidence before assuming a placebo effect could be clinically measurable on objective criteria as opposed to subjective criteria.

As for the third group, having just heard Dr Hall last eve at the Seattle Skeptics meeting speak on Snake Oil Science, the reason for the third group is to actually determine if there is a placebo effect or if improvement was part of the natural course of the ailment. I understand the concept that if a treatment is better than placebo it is better than the natural course of the ailment. But studies without the third arm may be contributing to poor conclusions about placebo effects, so it makes sense to include a no treatment group when feasible.

 

[fls]

I take issue with this claim. There is some evidence paying attention to things when doing research affects some outcomes, and there is evidence of a placebo effect on subjective assessments especially pain. But there is not good evidence that all treatments have a placebo effect. For example, you would want to see some good evidence before assuming a placebo effect could be clinically measurable on objective criteria as opposed to subjective criteria.

I don't think he's saying anything different than what you just said - that we expect to see changes in subjective assessments any time we provide a treatment.

As for the third group, having just heard Dr Hall last eve at the Seattle Skeptics meeting speak on Snake Oil Science, the reason for the third group is to actually determine if there is a placebo effect or if improvement was part of the natural course of the ailment. I understand the concept that if a treatment is better than placebo it is better than the natural course of the ailment. But studies without the third arm may be contributing to poor conclusions about placebo effects, so it makes sense to include a no treatment group when feasible.

That is what Billy Joe and I have been saying as well. If you are interested in addressing poor conclusions about placebo effects, a third arm may be useful, but it doesn't address poor conclusions about treatment effects. And the point was whether or not including a no treatment arm is feasible when it dissipates the power to draw conclusions about the treatment effect by reducing sample sizes.

Linda

 

[Skeptic Ginger]

I did say, 'when feasible'. An insufficient sample size would make it unfeasible.

But all treatments do not have a placebo effect. I think we may have overestimated the placebo effect. What was interesting in "Snake Oil Science" was the overview about research in general. For one, placebo was not initially compared to 'no treatment'. So the natural course of an illness was not always considered and may have led to the overrating of placebo effects. I found it especially telling in the book that so few research results were published that found 'no effect'. I think the book should be required reading for all medical practitioners.

Dr Hall noted a change in how we view evidence based medicine that I agree with. In fact, I need to add it to my sig.

Science based medicine should replace evidence based medicine. SBC is EVB + critical thinking.

 

[BillyJoe]

Science based medicine should replace evidence based medicine. SBC is EVB + critical thinking.

We always seem to end up saying the same thing, even when we start off seeming to say the exact opposite.


But didn't you mean SBM = EBM + critical thinking?

In any case, SBM would not be replacing EBM but adding to it.
However, my understanding is that the other factor is "plausibility":

SBM = EBM + plausibility

But perhaps we are saying the same thing.

 

[Skeptic Ginger]

Yes, I did mean SBM = EBM + critical thinking. I can't figure out how I came up with those dyslexic letters and worse, how I didn't notice. Brains are really weird things sometimes.

The point wasn't so much that one was replacing evidence based medicine, it was more a comment on the fact that evidence based medicine was not enough without a critical evaluation of the evidence you were looking at. It really is interesting once you start to notice how much supposed evidence really isn't. I was watching a couple things today, one was an acupuncture lecture I've heard before given by some doctor at the U of W. She repeats all sorts of things claiming research and her experience back the conclusions up. But some of the stuff I have personally spent considerable time investigating and do not believe the evidence to be anywhere near as reliable as she was claiming.

Yet if you were not to know that, you would come away from this lecture believing, for example, there really was acupuncture anesthesia being practiced on a regular basis in Cuba and China. There was one well documented case, but it was a single exceptional individual. The rest of the cases actually involve medication as well as the acupuncture according to what I found in my review of the literature. It really makes you start to question a lot of the evidence you thought was more reliable than perhaps it really is.

2) Acupuncture Today:

A brief history of acupuncture, acupuncture research, and the current evidence regarding the efficacy of acupuncture treatment. Also discussed is the epidemiology of acupuncture in the United States, as well as the training of acupuncturists, the different styles of acupuncture, the challenges facing high-quality acupuncture research, and a review of the evidence behind acupuncture for the treatment of morning sickness and fibromyalgia. Produced May 21, 2003.

Speaker: Nassim P. Assefi, M.D., Women's Health Specialist, Acting Instructor, Depts. of General Internal Medicine & OB/GYN, Harborview Medical Center, UW School of Medicine

I can't find an online viewing but here is a study she published.

A Randomized Clinical Trial of Acupuncture Compared with Sham Acupuncture in Fibromyalgia

We know there have been considerable successes with medical interventions. So it isn't that we are still using the equivalent of bloodletting or anything. But there is just so much research that is unsound.

The other thing I was watching today was a totally hokey program on the SciFi channel on crystal skulls. It was made to take advantage of the publicity from the new Indiana Jones movie. They spouted one discredited claim after another. Some idiot was claiming crystals grew like organic cells grew. They put Hoagland on as if he were some actual expert. He's the guy who can't get over the fact the face on Mars isn't one. And only once in the entire program did they even bother to have someone on who actually presented any science, then as usual, they brushed on by that as if it was just someone else's claims vs the quack's claims. No wonder the public thinks there is evidence for so much nonsense.

 

[fls]

I did say, 'when feasible'. An insufficient sample size would make it unfeasible.

Yup. Like I said, I think you are saying the same thing as Billy Joe - no need to take issue.

But all treatments do not have a placebo effect.

If 'placebo effect' is taken to be 'any improvements seen in the placebo control group', I think that it would be difficult to find a situation where no improvements were found. Especially for CAM studies where the outcomes are often subjective assessments and the conditions are usually of a type that wax and wane. I think it's useful to stress what we expect to see in the absence of any specific treatment effect - that we still expect to see improvement, sometimes a great deal of improvement (I think that's what his statement was intended to convey).

I think we may have overestimated the placebo effect.

In some ways, it's both overestimated and underestimated. I think the degree to which the improvements seen in the placebo group are due to a specific effect from the placebo have been overestimated. I realized at the time that a lot of Mind/Body medicine was over-blown, but even I was surprised by the review in the NEJM that showed no differences on objective measures. In a way, even Bausell over-emphasizes the effect by devoting two chapters in his book to physiologic mechanisms that probably aren't even relevant in most cases.

And I think the degree to which no treatment and expectation lead to apparent improvement is grossly underestimated. We get presented with story after story in support of CAM that "cannot be accounted for by placebo", that can easily be accounted for by placebo.

What was interesting in "Snake Oil Science" was the overview about research in general. For one, placebo was not initially compared to 'no treatment'. So the natural course of an illness was not always considered and may have led to the overrating of placebo effects. I found it especially telling in the book that so few research results were published that found 'no effect'. I think the book should be required reading for all medical practitioners.

That might be a good idea. At least in the US, the lack of condemnation of CAM (if not support) among medical practitioners gives it a bit of a foothold. I think that if there were more of a general understanding of why it should be condemned, it would help remove that air of legitimacy.

I was going to say that it would be better as required reading for CAM practitioners, but I suspect they have too much invested in the idea and would be forced to reject the information.

Dr Hall noted a change in how we view evidence based medicine that I agree with. In fact, I need to add it to my sig.

Science based medicine should replace evidence based medicine. SBC is EVB + critical thinking.

You and Dr. Hall seem to have been taught a different evidence-based medicine than I was taught. It may have been my proximity to the horse's mouth, I suppose.

Linda

 

[Don At Work]

Not really, at least in behavioral science. In some cases n = 50 in each group will yield a statistically significant, but practically useless result. In a between groups design, it is felt that at least 10 subjects in a group are needed to demonstrate that a robust effect will yield statistical significance. With a weaker effect, maybe n = 20 in a group might be needed.
The ns do not always justify the means.

I now hate you

 

[BillyJoe]

I can't seem to find the reference, but it is something I came across recently, an article perhaps, which stated that we should not be thinking in terms of EBM but SBM and the definition was actually SBM = evidence X plausibility, which actually comes back to the aphorism "extraordinary (implausible) claims require extraordinary evidence" or, in other words, the more implausible the claim of a particular treatment, for example homoeopathy, the more evidence is required before it is acceptable as a medical treatment. Most alternative medical therapies scrape the barrell on plausibility and, at best, just barely pass muster on evidence.

 

[Soapy Sam]

Non-detailed poll on the subject of definitions:-
If I take a long, sterile, stainless steel needle, shove it into someone's heart and waggle it around, there is a fair chance it would be murder.

But would it be accupuncture?

No. Look up meridians, which have no relationship to actual body parts as known to modern medical science.

I agree, and that reinforces a point I (and others) made early in the thread. Sticking needles in to people is not the same as acupuncture. The study in question did not test acupuncture, but rather needling (for want of a better word. There aren't any studies that have shown that acupuncture is effective, although needling may be.

Linda

I also agree. I am aware that many mainstream (NHS) physiotherapists say "accupuncure does have effect". I maintain it does not, as what they do is not accupuncture but "Dry needling". But the distinction is trivial to the public generally, who see anything involving needles and not an injection as "accupuncture"

ETA- My apologies for wierd spell/ typing errors in my posts of late. I acquired a wifi keyboard which is more wi than fi.

 

[Acleron]

I also agree. I am aware that many mainstream (NHS) physiotherapists say "accupuncure does have effect". I maintain it does not, as what they do is not accupuncture but "Dry needling". But the distinction is trivial to the public generally, who see anything involving needles and not an injection as "accupuncture"

ETA- My apologies for wierd spell/ typing errors in my posts of late. I acquired a wifi keyboard which is more wi than fi.

It's bewildering that people will accept without any evidence that sticking a sharp object through the skin at random does any good and yet injecting a vaccine which has been heavily tested causes harm.

BTW I before E except after C, it's a weird rule.

 

[Skeptic Ginger]

...
You and Dr. Hall seem to have been taught a different evidence-based medicine than I was taught. It may have been my proximity to the horse's mouth, I suppose.

Linda

It's not what either of us were taught. It is what we are changing to.

I realize we are close to agreement here. But a section of the Skepdic entry on placebo effect is worth looking at because it addresses the issues better than I can. It also included comments about the NEJM article I believe you were referring to.

What the reanalysis shows is that there are a number of factors that can affect many treatments and the evaluation of those treatments, making it very difficult to be sure just what it is about an intervention that produces improvement or perceived improvement. We must also consider "artifacts such as the natural history of a disease (that is, the tendency for people to get better or worse during the course of an illness irrespective of any treatment at all), the fact that people behave differently when they are participating in an experiment than when they are not, a desire to please the experimental staff by providing socially desirable answers..." (Bausell 2007: 27) and a host of other factors unrelated to the pill we are administering and independently of any mechanism that we believe is producing any observed effects.

In May 2001, The New England Journal of Medicine published an article that called into question the validity of the placebo effect. "Is the Placebo Powerless? An Analysis of Clinical Trials Comparing Placebo with No Treatment" by Danish researchers Asbøjrn Hróbjartsson and Peter C. Götzsche "found little evidence in general that placebos had powerful clinical effects." Their meta-analysis of 114 studies found that "compared with no treatment, placebo had no significant effect on binary outcomes, regardless of whether these outcomes were subjective or objective. For the trials with continuous outcomes, placebo had a beneficial effect, but the effect decreased with increasing sample size, indicating a possible bias related to the effects of small trials." (Most of the studies evaluated by Hrobjartsson and Götzsche were small: for 82 of the studies the median size was 27 and for the other 32 studies the median was 51.)

"The high levels of placebo effect which have been repeatedly reported in many articles, in our mind are the result of flawed research methodology," said Dr. Hrobjartsson, professor of medical philosophy and research methodology at the University of Copenhagen.*

Typical of the kind of flawed research methodology Hrobjartsson is referring to would be that of surgeon J. Bruce Moseley who performed fake knee surgery on eight of ten patients. (Fake surgery involves making an incision on the knee and stitching it up.) Six months after the surgery all the patients were satisfied customers. Rather than conclude that the patients didn't need surgery or that the surgery was useless because in time the patients would have healed on their own, he and others concluded that the healing of the eight who did not have surgery was due to the placebo effect, while the two who had real surgery were better because of having had the operation. Irving Kirsch and Guy Sapirstein have been accused of making the same kind of methodological error in their controversial meta-analysis that found that anti-depressants work by the placebo effect, rather than that anti-depressants are unnecessary and useless.

One more example should suffice to make the point that better designs of placebo studies are needed.

Forty years ago, a young Seattle cardiologist named Leonard Cobb conducted a unique trial of a procedure then commonly used for angina, in which doctors made small incisions in the chest and tied knots in two arteries to try to increase blood flow to the heart. It was a popular technique—90 percent of patients reported that it helped—but when Cobb compared it with placebo surgery in which he made incisions but did not tie off the arteries, the sham operations proved just as successful. The procedure, known as internal mammary ligation, was soon abandoned ("The Placebo Prescription" by Margaret Talbot, New York Times Magazine, January 9, 2000).*

Did Cobb show that this kind of surgery works by the placebo effect? Or did he show that the surgery was unnecessary because most of the patients would have healed on their own if nothing had been done?

To rule out the natural history of a disease or regression to the mean, many researchers have used a third control group—those who receive no treatment at all. If the placebo group shows better results than the group getting nothing, then surely the placebo is effective. Hrobjartsson and Götzsche think most of these studies, too, are flawed, mainly due to having samples that are too small or due to patients who make reports aimed at pleasing the researcher.

The article goes on to discuss this further including quotes from Bausell's book.

 

[Skeptic Ginger]

The thing I found most memorable about review of the Snake Oil Science topic was just how few the "failed to find any results" studies end up being published. That must be having an affect on those meta-analyses.

 

[fls]

It's not what either of us were taught. It is what we are changing to.

That's what I mean. You seem to be changing to what I was taught in the first place.

I realize we are close to agreement here. But a section of the Skepdic entry on placebo effect is worth looking at because it addresses the issues better than I can. It also included comments about the NEJM article I believe you were referring to.

The article goes on to discuss this further including quotes from Bausell's book.

I agree that it is worthwhile looking at the components of the placebo effect in greater detail, especially because (as you are well aware) it gets presented as a valuable entity in its own right. And I've made a point of trying to counteract the impression given in the examples you quoted (about knee surgery, internal mammary ligation and anti-depressants) that one has to do 'something' in order to enjoy the placebo effect. I wasn't trying to say that, just because it may not be feasible to test multiple hypotheses in this acupuncture study, any one of the multiple hypotheses is not worth studying in its own right.

I just think it deserves better investigation than throwing in an untreated control group as almost an afterthought when doing a regular clinical trial - something more along the lines of the study on placebo in Irritable Bowel Syndrome. And that we already have a plethora of clinical trials weakened by throwing in extraneous hypotheses for testing - especially in CAM research.

Even if someone doesn't agree with those suggestions, at the very least, I don't think I deserve to be treated as a bit dense for making them (not by you, but by Sherman Bay (since you leapt into the middle of my defense of myself (and Billy Joe))).

Linda

 

[Skeptic Ginger]

I'm confused Linda. Are you saying that having a study arm which identifies the natural course of an illness with no intervention is not of much value?

 

[fls]

I'm confused Linda. Are you saying that having a study arm which identifies the natural course of an illness with no intervention is not of much value?

I said that it can be of value.

Other things that I think are of value:

Clinical trials that have sufficient power to draw valid conclusions.
Experiments that distinguish between which components of the placebo effect are due to intervention (and the nature of that intervention) and which are not.

Things that I think are of lesser value:

Clinical trials which have inadequate power to draw valid conclusions, or have sufficient power to draw only trivial conclusions.
Experiments that merely repeat well-established hypotheses about the placebo effect and cannot further our understanding.

Linda

 

[fls]

I'm confused Linda. Are you saying that having a study arm which identifies the natural course of an illness with no intervention is not of much value?

I'll put the question to you...

What did the untreated control group, in the acupuncture study we have been talking about, tell you about the placebo effect that we don't already know?

Linda