Acupuncture appears to show results.

[Ivor the Engineer]

<snip>

Another practical reason which springs to mind is that it would probably be difficult to recruit and retain a non-treatment group if the study period was more than a day or so.

I don't understand this.

I was thinking that people may be a bit disgruntled to find themselves in the no-treatment group, and thus be unwilling to take part if the duration they are expected to delay treatment by is significant.

 

[fls]

I was thinking that people may be a bit disgruntled to find themselves in the no-treatment group, and thus be unwilling to take part if the duration they are expected to delay treatment by is significant.

Now I understand.

Yeah, there are all kinds of tricks. There's usually a delay between enrollment and the start of treatment anyway, so sometimes you can hide an extra 4-8 weeks in there. You don't tell the people they're in the no-treatment group, you tell them they're on the waiting list for the treatment group. You lie (in a completely ethical way, of course ) about what the treatment consists of.

Linda

 

[BillyJoe]

If both true and sham acupuncture show similar results, what is the conclusion? You still don't know if acupuncture works or not...It can be that both are placebos, or that both are beneficial due to the needle sticking, or something else.

But we are testing for the specific effect* of acupuncture (True Acupuncture), not the effect of acupuncture as a needle sticking procedure (Sham Acupuncture). Sham Acupuncture is therefore the correct control to use, and the only control you need to use.

*The specific effect of acupuncture is the idea that putting needles in at specified points for a particular disease or symptom helps cure that disease or resolve that symptom.

 

[Skeptic Ginger]

When doctors go on strike, the death rate goes down.

It's important to find out what happens when you do nothing.

That is fascinating. I can't wait for a follow up analysis.

 

[Skeptic Ginger]

When you say "placebo effect", what exactly are you talking about?

More than just observer bias. It isn't that I doubt placebo effects, what I doubt it how good is our actual evidence on the strength of the effect? How much of the evidence was accumulated without a no intervention arm?

What did the untreated control group in the acupuncture study tell us, and how does it make this less of an issue?

Linda

In the OP study how do you explain such a huge decrease in migraine events in the untreated group?

I think it told us there were some variables here like grossly inaccurate patient recall of baseline events that needed to be sorted out before we started claiming the statistical differences in outcomes were significant.

 

[Skeptic Ginger]

I may need to revisit the acupuncture evidence one of these days but I did look into it extensively not too long ago. There seemed to be a false belief that there was all this supporting evidence for its effects. But I recall a lot of the evidence came out of China and was unreliable. For example there were a lot of claims of using acupuncture anesthesia when in reality drugs were also used.

Going on 9 pages, I am probably repeating myself here. Anyway, I am particularly suspicious of the research in this treatment because of the information I uncovered reviewing the data.

 

[BillyJoe]

In India, where the slog work is left to junior assistants who try to please their masters, the postive outcomes for acupuncture are 90% compared with 10% in the west.

Anyway, skeptigirl, we are trying to get some agreement about how to conduct a proper trial of acupuncture. The suggestion is that you need only two groups: True Acupuncture and Sham Acupuncture. And to maximise the power of the trials, where there are likely to be a very limited number of patients enrolled, no other groups should be used.

regards,
BillyJoe

 

[fls]

More than just observer bias. It isn't that I doubt placebo effects, what I doubt it how good is our actual evidence on the strength of the effect? How much of the evidence was accumulated without a no intervention arm?

If you haven't already done so, you should read this article in its entirety.

http://content.nejm.org/cgi/content/short/344/21/1594

In the OP study how do you explain such a huge decrease in migraine events in the untreated group?

Easy peasy (didn't you claim to have read Snake Oil Science? ).

Regression to the mean probably accounts for the biggest chunk. Migraines wax and wane. People are going to show up at the doctor's office and agree to enroll in a study when it is waxing, not when it is waning. By the end of the study, they are more likely to be closer to the mean than further away.

Changes in behaviour as a result of paying increased attention to their symptoms.

Desire to please the investigators.

The use of anti-migraine medication.

I think it told us there were some variables here like grossly inaccurate patient recall of baseline events that needed to be sorted out before we started claiming the statistical differences in outcomes were significant.

Do you think patient recall of baseline events differed between groups in a way that was associated with the assignment of groups? Because otherwise it's irrelvant.

Linda

 

[Skeptic Ginger]

If you haven't already done so, you should read this article in its entirety.

http://content.nejm.org/cgi/content/short/344/21/1594

Just did. Don't see anything in there I didn't already think was in there. Was there something in particular you were thinking of?

Easy peasy (didn't you claim to have read Snake Oil Science? ).

Regression to the mean probably accounts for the biggest chunk. Migraines wax and wane. People are going to show up at the doctor's office and agree to enroll in a study when it is waxing, not when it is waning. By the end of the study, they are more likely to be closer to the mean than further away.

Changes in behaviour as a result of paying increased attention to their symptoms.

Desire to please the investigators.

The use of anti-migraine medication.



Do you think patient recall of baseline events differed between groups in a way that was associated with the assignment of groups? Because otherwise it's irrelvant.

Linda

An individual patient with a chronic condition that flared sending the person to seek medical care would regress toward the mean. Were the subjects all on the same circadian cycle or something? Wouldn't some of them get better and some get worse? Otherwise what does that say about the adequacy of the sample size?

Perhaps I missed something and these folks were entered into the study one at a time. I thought they were entered into the study at the same time. I put the article away so I'll have to dig it out and recheck.

I agree that randomizing would fix the poor historian issue, but I think the number of headache events really changes the sample size. If I have patients who have one headache a month and I only carry the study out for 2 months that can affect the actual sample size effect.

There are lots of studies with these little marginal results for acupuncture and other CAMs. But when you repeat the study you get a different outcome. It's like the cluster effect in studying rare cancer events. 5 people getting brain cancer in a year who work in the same building is not by itself, evidence the building is the source of the cancer even though the rate of brain cancer would definitely be statistically significant.

(BTW, I haven't read the book. I heard an hour talk on it from The SkepDoc, Harriet Hall.)

 

[Skeptic Ginger]

In India, where the slog work is left to junior assistants who try to please their masters, the postive outcomes for acupuncture are 90% compared with 10% in the west.

Anyway, skeptigirl, we are trying to get some agreement about how to conduct a proper trial of acupuncture. The suggestion is that you need only two groups: True Acupuncture and Sham Acupuncture. And to maximise the power of the trials, where there are likely to be a very limited number of patients enrolled, no other groups should be used.

regards,
BillyJoe

One issue is the difference in scientists interpreting research and the woo believers having their beliefs reinforced by CAM studies. Without the "no treatment" arm you get people claiming the sham acupuncture must have worked for mysterious reasons.

Another issue is the one I am referring to in the OP article. If we didn't have that no treatment arm we wouldn't know that there was such a large change in the no treatment group. It affects the statistical analysis of the outcome if there is a different denominator of baseline migraine events.

 

[BillyJoe]

One issue is the difference in scientists interpreting research and the woo believers having their beliefs reinforced by CAM studies. Without the "no treatment" arm you get people claiming the sham acupuncture must have worked for mysterious reasons.

I don't get that.
True acupuncture is no better than sham acupuncture.
What could be clearer than that!

Give them a few more groups and insufficient numbers is each group and the woos will have a field day with the results.

Another issue is the one I am referring to in the OP article. If we didn't have that no treatment arm we wouldn't know that there was such a large change in the no treatment group. It affects the statistical analysis of the outcome if there is a different denominator of baseline migraine events.

So what?
True acupuncture is no better than sham acupuncture.
Period.

 

[Ivor the Engineer]

<snip>

So what?
True acupuncture is no better than sham acupuncture.
Period.

"Scientists find new energy meridians!"



(Never underestimate the power of stupid people in large numbers.)

 

[fls]

Just did. Don't see anything in there I didn't already think was in there. Was there something in particular you were thinking of?

Just that it answered these questions...

"It isn't that I doubt placebo effects, what I doubt it how good is our actual evidence on the strength of the effect? How much of the evidence was accumulated without a no intervention arm?"

An individual patient with a chronic condition that flared sending the person to seek medical care would regress toward the mean. Were the subjects all on the same circadian cycle or something? Wouldn't some of them get better and some get worse?

I'm suggesting that since they were more likely than not to be worse at the time of enrollment, they would be more likely than not to be better at the time of subsequent measurements. Unless an inordinate period of time passed between enrollment and the start of the study (the description suggests not).

Otherwise what does that say about the adequacy of the sample size?

So now we are in agreement that adequate sample size is relevant?

Perhaps I missed something and these folks were entered into the study one at a time. I thought they were entered into the study at the same time. I put the article away so I'll have to dig it out and recheck.

Unless you think many months passed between the time of enrollment and the time the study started, does this matter? (As far as I can tell, they were entered into the study at the same time.)

If there was a very long lead-in period to the study (in which case they have an amazing retention rate!), then I would suggest instead that most of the improvement was because of the use of a novel (to at least some of them, if not all?) drug.

I agree that randomizing would fix the poor historian issue, but I think the number of headache events really changes the sample size. If I have patients who have one headache a month and I only carry the study out for 2 months that can affect the actual sample size effect.

Yes.

There are lots of studies with these little marginal results for acupuncture and other CAMs. But when you repeat the study you get a different outcome. It's like the cluster effect in studying rare cancer events. 5 people getting brain cancer in a year who work in the same building is not by itself, evidence the building is the source of the cancer even though the rate of brain cancer would definitely be statistically significant.

Yes. Which is why Billy Joe and I suggested that it would be better to design studies that wouldn't return marginal results. And I am still wondering why we're getting so much flak for making that suggestion.

(BTW, I haven't read the book. I heard an hour talk on it from The SkepDoc, Harriet Hall.)

Ah. You gave a very different impression in this post.

Linda

 

[fls]

One issue is the difference in scientists interpreting research and the woo believers having their beliefs reinforced by CAM studies. Without the "no treatment" arm you get people claiming the sham acupuncture must have worked for mysterious reasons.

If people are ignoring evidence in order to draw conclusions that are not valid, is there much point in adding to the evidence to be ignored?

Another issue is the one I am referring to in the OP article. If we didn't have that no treatment arm we wouldn't know that there was such a large change in the no treatment group. It affects the statistical analysis of the outcome if there is a different denominator of baseline migraine events.

It doesn't change the statistical analysis of the outcome. The denominator comes from the treatment groups, not from the no treatment group.

Linda

 

[Skeptic Ginger]

Just that it answered these questions...

"It isn't that I doubt placebo effects, what I doubt it how good is our actual evidence on the strength of the effect? How much of the evidence was accumulated without a no intervention arm?"

I realize it did. I had already thought the evidence was that the effect of placebos was frequently overstated. But we also have a number of studies which dispel many other believed claims such as the effect of vitamin C on URIs. That doesn't stop people including many in the medical profession from continuing to hold to the myths.

In other words, it isn't that we don't know the answer, it's that we may need to demonstrate it many more times to counter years of believing placebo effects were dramatic most of the time.

So now we are in agreement that adequate sample size is relevant?

Why would I not agree with this?

Yes. Which is why Billy Joe and I suggested that it would be better to design studies that wouldn't return marginal results. And I am still wondering why we're getting so much flak for making that suggestion.

Maybe because you are misinterpreting the point I am making and think it is a debate with you when it isn't. I may be merely emphasizing other priorities in CAM research that you are not considering in this single study. It is not a disagreement, merely a different set of issues being focused on.

Ah. You gave a very different impression in this post.

Linda

Dr Hall gave a sufficient review of the book and I also know what it is about from its reputation. This is stuff I have known about for years. I don't need to read more on this subject. l do think other people need to read more on it. Do you think most medical providers are as skeptical about research as the research suggests they should be? Judging from the acupuncture lecture I posted a link to, it is still not common knowledge among practicing providers. Perhaps I should have qualified my enthusiasm and said everyone who isn't familiar with this stuff should read the book.

I think evidence based medicine and the gold standard of randomized clinical trials has become the new paradigm in medicine. But what I think is very often either forgotten, not considered, or in some cases not actually known is how much of what we do, or think we know is based on very weak evidence simply because the research has not been done, or because the research cited over and over has not really been critically reviewed even when published in peer reviewed journals.

How many of those peer reviews are done by professionals who are aware that almost no negative results are published from Asian countries? How many reviewers know the overall characteristics of the body of medical research and that it is biased in some general ways we don't often think about, such as the small number of failed hypotheses that ever get published?

When I looked into the acupuncture research (as I mentioned, sorry I am repeating myself) I found that there is a general belief that research exists showing such things as its effectiveness as an anesthetic. But what exists in fact, is a lot of very flawed research. Yet the doctor in the lecture spoke of the body of research on acupuncture as if it were fairly substantial. It may be substantial in volume, but certainly not in reliably interpreted conclusions.

How much 'knowledge' are we repeating because we heard it from someone we expected to have accurate information? But they are repeating it and we go on to repeat it. You were in that discussion we had where we looked for the data the measles morbidity and mortality rates came from. You could barely get back to the raw data. Every source cited non-original sources.

I think that is the case with the acupuncture research. Let me give you two scenarios. In one scenario we start with the conclusion that in general there is a lot of evidence acupuncture treatments have an effect, placebo or not, an effect occurs. In the other scenario we start with the conclusion that the placebo effect itself has been exaggerated and the studies on acupuncture which have all been properly done have failed to show much results at all, again, whether the effects were placebo or not is a separate issue for the moment.

Now considering those two different premises just how significant are the results of this single study? If you used the former premise you might think, maybe the proper Qi diagnosis (or whatever you call it) and the choice of acupuncture points by the traditional Chinese methods did matter and perhaps we need to reconsider how we are studying acupuncture. If you used the the latter premise, you would likely find these results interesting, but by no means convincing.

I look at this study using the latter premise. I see no existing body of literature suggesting that acupuncture, including a placebo mechanism of action, has a significant effect. Getting a statistically significant outcome in a single study is just not convincing. And the diagnosis and choice of puncture points is one of those extraordinary claims that needs extraordinary evidence, which this study is not. I am not discounting the possibility that acupuncture works, but given all the failed results in previous studies, given the assumption a lot of research that found no significant effect was subsequently never published, and, considering the claim is extraordinary that Qi exists and there is some well defined rationale of choosing acupuncture points, I am not impressed by this study.

We don't disagree and I am not objecting to what you've posted as far as the study goes. But I do think we are looking through two different lenses as to what is most in focus. You are just more focused on this individual study and I am more focused on the general problems with the studies on acupuncture over all. Neither focus is wrong.

 

[Skeptic Ginger]

Quackwatch on acupuncture

Harriet Hall, a retired family practitioner who is interested in quackery, has summed up the significance of acupuncture research in an interesting way:

Acupuncture studies have shown that it makes no difference where you put the needles. Or whether you use needles or just pretend to use needles (as long as the subject believes you used them). Many acupuncture researchers are doing what I call Tooth Fairy science: measuring how much money is left under the pillow without bothering to ask if the Tooth Fairy is real.

As for the existing body of research and extraordinary claims:

Although the report described some serious problems, it failed to place them into proper perspective. The panel acknowledged that "the vast majority of papers studying acupuncture consist of case reports, case series, or intervention studies with designs inadequate to assess efficacy" and that "relatively few" high-quality controlled trials have been published about acupuncture's effects. But it reported that "the World Health Organization has listed more than 40 [conditions] for which [acupuncture] may be indicated." This sentence should have been followed by a statement that the list was not valid.

Far more serious, although the consensus report touched on Chinese acupuncture theory, it failed to point out the danger and economic waste involved in going to practitioners who can't make appropriate diagnoses. The report noted:

* The general theory of acupuncture is based on the premise that there are patterns of energy flow (Qi) through the body that are essential for health. Disruptions of this flow are believed to be responsible for disease. The acupuncturist can correct imbalances of flow at identifiable points close to the skin.
* Acupuncture focuses on a holistic, energy-based approach to the patient rather than a disease-oriented diagnostic and treatment model.
* Despite considerable efforts to understand the anatomy and physiology of the "acupuncture points," the definition and characterization of these points remains controversial. Even more elusive is the scientific basis of some of the key traditional Eastern medical concepts such as the circulation of Qi, the meridian system, and the five phases theory, which are difficult to reconcile with contemporary biomedical information but continue to play an important role in the evaluation of patients and the formulation of treatment in acupuncture.

Simply stated, this means that if you go to a practitioner who practices traditional Chinese medicine, you are unlikely to be properly diagnosed. Very few publications have mentioned this, which strikes me as very strange. Even Consumer Reports magazine has advised readers who want acupuncture treatment to consult a practitioner who is NCCAOM-certified. I advise people to avoid "certified" practitioners. Because the training needed for certification is based on nonsensical TCM theories, the safest way to obtain acupuncture is from a medical doctor who does research at a university-based medical school and does not expouse such theories.

 

[fls]

I realize it did. I had already thought the evidence was that the effect of placebos was frequently overstated. But we also have a number of studies which dispel many other believed claims such as the effect of vitamin C on URIs. That doesn't stop people including many in the medical profession from continuing to hold to the myths.

In other words, it isn't that we don't know the answer, it's that we may need to demonstrate it many more times to counter years of believing placebo effects were dramatic most of the time.

That used to be my assumption as well - that all we need to do is accumulate information past the point of redundancy in order to counteract unsupported claims. But as I've gained experience in this area, my observations don't support this assumption. It seems to take an inordinate amount of information to change attitudes and subsequent behaviour, whereas changing attitudes by other means (or by using knowledge from different areas) means that a more reasonable amount of information is sufficient to change behaviour (that whole 'give a man a fish vs. teach a man to fish' thing).

Why would I not agree with this?

Maybe because you are misinterpreting the point I am making and think it is a debate with you when it isn't. I may be merely emphasizing other priorities in CAM research that you are not considering in this single study. It is not a disagreement, merely a different set of issues being focused on.

I understand that you are talking about addressing priorities (and I have been in agreement with those priorities). I disagree with you on some of the details of how to address those priorities.

I think evidence based medicine and the gold standard of randomized clinical trials has become the new paradigm in medicine. But what I think is very often either forgotten, not considered, or in some cases not actually known is how much of what we do, or think we know is based on very weak evidence simply because the research has not been done, or because the research cited over and over has not really been critically reviewed even when published in peer reviewed journals.

How many of those peer reviews are done by professionals who are aware that almost no negative results are published from Asian countries? How many reviewers know the overall characteristics of the body of medical research and that it is biased in some general ways we don't often think about, such as the small number of failed hypotheses that ever get published?

It is my impression that in the US, EBM is strongly associated with the Cochrane Collaboration - i.e. EBM is what is done at the Cochrane Collaboration. Is that an accurate impression?

I think that is the case with the acupuncture research. Let me give you two scenarios. In one scenario we start with the conclusion that in general there is a lot of evidence acupuncture treatments have an effect, placebo or not, an effect occurs. In the other scenario we start with the conclusion that the placebo effect itself has been exaggerated and the studies on acupuncture which have all been properly done have failed to show much results at all, again, whether the effects were placebo or not is a separate issue for the moment.

Now considering those two different premises just how significant are the results of this single study? If you used the former premise you might think, maybe the proper Qi diagnosis (or whatever you call it) and the choice of acupuncture points by the traditional Chinese methods did matter and perhaps we need to reconsider how we are studying acupuncture. If you used the the latter premise, you would likely find these results interesting, but by no means convincing.

I look at this study using the latter premise. I see no existing body of literature suggesting that acupuncture, including a placebo mechanism of action, has a significant effect. Getting a statistically significant outcome in a single study is just not convincing. And the diagnosis and choice of puncture points is one of those extraordinary claims that needs extraordinary evidence, which this study is not. I am not discounting the possibility that acupuncture works, but given all the failed results in previous studies, given the assumption a lot of research that found no significant effect was subsequently never published, and, considering the claim is extraordinary that Qi exists and there is some well defined rationale of choosing acupuncture points, I am not impressed by this study.

We don't disagree and I am not objecting to what you've posted as far as the study goes. But I do think we are looking through two different lenses as to what is most in focus. You are just more focused on this individual study and I am more focused on the general problems with the studies on acupuncture over all. Neither focus is wrong.

I am not focussed on this individual study. I'm not sure why you made this assumption considering that I have talked about the broad picture of acupuncture research starting with my very first post in this thread, and throughout I have made references to CAM research, and how this study illustrates problems found in both.

Your scenarios serve as an example of what I was talking about at the beginning. Changing the premise under which you approach this study (i.e. 'attitude') changes your interpretation, without making any changes to the information that already exists. I think it's of value to address those premises before adding to the information that already exists.

Linda

 

[Skeptic Ginger]

Well I see that from your 1st page posts going back, perhaps I became confused in your later discussion about the no treatment arm.

There is a lot of emphasis on the Cochrane reviews but they highlight the lack of RCTs among the large volume of research out there.

 

[JJM]

"The largest "randomized" acupuncture study ever done: Why did they even bother?"

"Another Worthless Acupuncture Study"

 

[Acleron]

"The largest "randomized" acupuncture study ever done: Why did they even bother?"

"Another Worthless Acupuncture Study"

Either they cannot do a properly controlled study because they lack the aptitude or they won't do such a trial because they fear the results.