Homeopathic research report for nitpick

[MRC_Hans]

Perhaps this should be in riddles, but...

Background: Kumar (aka Kayveeh) posted a link to this report over at otherhealth. The experiment uses Kirlian Photography (KP) try and find properties of homeopathic preparations. My immidiate reaction was to more or less brush off the whole thing, because trying to explain things to Kumar has long proved inefficiaous.

However, Neil Shere (aka Bach, aka BWV11) came in and asked me to make a nitpick of it. For some reason, Neil seems to have some respect for my ability to nitpick papers, even if he rarely agrees with me. Since Neil occasionally makes some sense, I complied.

I'm really quite sure of my take on the paper, but still a second opinion can never hurt, and also, we could take it as an exercise.

http://lkm.fri.uni-lj.si/xaigor/slo/znanclanki/instrumental.htm

I find one possible serious error, and one obvious fatal error.

Hans

 

[Graham Jackman]

I can't understand why they did this experiment at all and if you wanted to monitor the discharges, why not simply integrate them with a digital camera or use unexposed paper and then develop it. The major flaws that I can also see are that they don't provide us with standard errors of their measurements, so we can't evaluate the changes properly. In addition, they don't correct for the multiple measurements. If you make a number of tests on the same data you increase the chances of a statistical "hit". E.g. if you make 20 tests, the chances of finding a probability 0.05 are greatly enhanced.
Even if there was some reliability to the testing, why do this instead of a proper trial with preparations in patients?

 

[Zep]

Here's the only paragraph that matters in the whole report:

When the homeopathic dilutions D9, D16 and D23 were compared with the equally succussed and "diluted" (sham-potentised) pure water (K9, K16, K23), the results of comparison between D9 and K9 were approximately the same as when D9 was compared with the control water (K0).

Translation: Potentised homeopathic remedy at 9X is indistinguishable from water. The other comparisons were the same.

At D16 the values of “homeopathic” parameters varied with regard to whether D16 was compared with K16 or K0.

Translation: We could not find real differences between a 16X remedy and water.

The experiments comparing sham-potentised and control water showed some differences between K16 and K0 and no significant difference between K23 and K0.

Translation: Water = water.

Still, the results of the comparison between D23 and K0 were much more marked than those of D23 compared with K23.

Translation: We started with dirty water as our control.

 

[MRC_Hans]

Well, my take on it is the following:

The procedures are very good. They prepare remedies and controls from the same basic stock solvent (water), and they even dilute and succuss the controls following the same procedure as the remedy.

The differential measurement is also a very good idea. Of course, it is, to a large degree a mitigation of the problems stemming from their choice of a measuring metod (KP) that has, at best, an extremely high noise level.

The method of analyzing the KP recordings is also sensible, although, since no real knowledge exists on how to interpret KP, the chosen 14 parameters are essentially arbitrary.

Finally, the excecution of the protocol seems to be flawless.

Now, we come to the problems. First of all, the report contains a series of graphs. These seem to depict numeridal data, and it is noticeable that all graphs contain a set of reference bars (black), which are completely identical on all graphs. The only way I can interpret this is that they have used the absolute readings instead of the differential readings, and compered tham all to a single reference. I am ready to be corrected on this, however.

Then we come to the final data representation (T2) and the conclusion. In T2, which contains a summary of the data, they find that four of the parameters recorded for each test instance seem to show a pattern corresponding to the potentization of the samples. They perform a statistical significance analysis on these parmeters in isolation and find that there is indeed a pattern.

Finally, they conclude that they have shown that there is "on at least some of the parameters" a correlation with the potency, and they claim to have thus proven a difference between the remedy and the sham preparation.

I find that this treament of the data is invalid, and the conclusion is thus fatally wrong. Since all 14 parameters are recorded together for each set of samples, the task is to show that the differences, as a whole are not random (the null hypothesis). If you select a subset of parameters on the basis that they show a pattern, you have defeated the null hypothesis, and a statistical significance analysis is invalid, because it will of course confirm that there is a pattern (since you selected for one).

Thus, the conclusion of the paper is based on data mining and must be rejected.

As aminor point, I observe that they do note that the readings on the samples that did actually contain some substance is non-siginificant. Since we might expect that those should show some difference, they ought to have discussed the implication or this on the evaluation of the measurement method. I think that the lack of significant readings from samples that had a factual difference indicate that the method is not useful for the purpose.



Hans

 

[MRC_Hans]

Oh, my! This has turned into the never ending story. For some reason Neil is seriously enamoured to this paper, and he is defending it with all his might. .. Which doesn't amount to much.

If you have no drying paint to watch, look here:

http://www.hpathy.com/homeopathyforums/forum_posts.asp?TID=6629&PN=1&TPN=6

Hans

 

[MRC_Hans]

Heheh, My friend Neil has holed up and declared he will need a couple of weeks to compose a refutation. I will now make a prediction:

It will be a long-winded rant containing references to at least one of the following: Darwin, golf results, musical instruments.

It will contain several more or less overt attacks on my integrety.

It will repeat that the report is correctly made and that data mining is OK.

It will not address any of the central points.

Hans