drkitten said:
First, there's no way one can know beforehand how large a sample size will be necessary before running the study, since the size of the sample necessarily interacts with the size of the effects.
Huh?
This is just simply not true. There are statistics built into the study as it's being designed. You can determine before you even start the study exactly how many people must meet the study endpoint in order for a statistically significant effect is observed. Statisticians know
exactly how many patients need to be in the study to show an effect before the study is ever started. Inadequate statistics can render the study meaningless.
In addition, problems begin to arise, among many other reasons, when the study is not properly insulated from bias (e.g., unblinded, unbalanced control groups, etc.), a "shakey" or clinically meaningless endpoint is used, or post-hoc analysis of a non-primary endpoint shows a positive result. These are what good scientists use to determine overall validity of a study.
drkitten said:
The second reason for running an experiment with small groups is that results can still be useful, even if they don't hit "significance."
The main reason, actually, why this is done is due to cost considerations - almost nothing else. People rarely (if ever) draw conclusions from small studies.
drkitten said:
A difference this large (and a probability of < 0.10) suggests that there is likely to be some effect when studied in more detail, and as such can be used as a club to beat money for a larger-scale study out of the appropriate funding agencies.
A p value of anything other than <0.05 is meaningless, even in a small study. Again, if you are doing a "proof of concept" using this study, then you run a larger confirmatory study that is sufficiently powered to detect a difference. In subjective studies where patient's report whether or not their symptoms are improved, and there is not a clear-cut time where they are either better or worse other than their subjective reporting, you need very large numbers to show a treatment effect.
drkitten said:
On the other hand, if 18/22 of the experimental group (or worse, 22/22 of the experimental group) had displayed symptoms, then we'd have evidence that echinacea has no effect whatsoever, for much less than the cost of a full-fledged study involving hundreds of participants.
The study was not powered to determine a "non-effect". And, as I mentioned above, the subjectivity of patient reporting necessitates much larger numbers of patients to show clinical relevance. In other words, this study is meaningless.
There is ample evidence from
literally hundreds of other studies that show that ecchinacea has, at worst, no observable effect and, at best, minimal impact in the amelioration of cold symptoms, duration, etc. In my opinion, there is no need to waste additional money investigating and/or buying this herb. The results are not equivocal.
-TT
