Ed said:
Ask for references of double blind studies.
I feel I ought to say something about the "randomised, double-blind, placebo-controlled study" (RDBPCT for short). It's vital to understand that this isn't the standard test that every medicine must pass in order to be accepted - far from it. It is in fact the last resort "magnifying glass" study to try to figure whether there's anything there at all in cases where any effect is so tiny that its very existence is in doubt.
Unfortunately virtually all "alternative medicine" is in this category.
Thus, when I see a paper title including these words, I know before I even read it that the subject is probably some sort of weird woo-woo which is being gleefully hyped by the "natural medicine" brigade, but which has so little real effect that nobody's really sure whether it has any at all. So, the very fact that a RDBPCT has been done in the first place raises my level of suspicion that there's likely to be precious little effect there (if any). You WON'T find such trials in the literature for most "real" medicine, even recently introduced drugs, for the very simple fact that they are impractical, unnecessary and unethical.
Consider. Placebo control. That means half your patient group gets nothing. Where there is already an existing treatment with proven efficacy (as there usually is) there's no way any ethics committee would allow that. What is really required is a comparison with existing best practice, not with nothing at all.
Double blind. That means that neither the researcher nor the person looking after the animal knows which has been given. Often totally impractical for a comparison to an existing drug. What if the standard treatment is an injection, but the new drug is a pill? Most of the time, it's simply impossible to dress the things up as being the same.
Randomised. Actually, do we NEED more data on the existing treatment? It's almost certainly been studied
ad nauseam already.
In fact what usually happens is that something which looks promising in preliminary work (rational mode of action, good results in
in vitro testing, experimental animal testing and so on) is finally judged as ready to be tried on patients. A group of volunteer patients is recruited, given full information about what's going on (or their owners in the case of animal patients), and given the new drug. They are monitored very closely indeed, and the performance of the new drug compared to the already-known performance of the existing drug.
For example, a new drug introduced recently to canine medicine is trilostane, for the treatment of something called Cushing's disease. The previous treatment, mitotane, had a very powerful effect and often worked well, but it had a poor safety margin and it was unlicensed. Trilostane was assessed in a group of suitable patients and found to perform comparably to mitotane so far as clinical response was concerned, and with a much better safety margin. On this basis, a product licence was granted. But you certainly won't find a RDBPCT study assessing trilostane!
(As an aside, this drug works so well that the vet who was featured on the famous
Horizon programme extolling the virtues of homoeopathic treatment for Cushing's disease, told me last week that now when people come to him looking for homoeopathy for this condition, he recommends trilostane instead!)
In contrast, a paper was published recently looking at a herbal remedy for arthritis in the dog. The study compared the remedy to a placebo, and in order to satisfy ethical considerations, scope had to be allowed for the owner administering a standard painkiller as well if they thought the dogs were still in pain. This of course did muddy the water to some extent, but the powers that be required it.
Several different measurements were made, some subjective, some objective. The only clear and obvious finding was that the herbal remedy made the dogs smell funny. As a result the investigators were unblinded - they knew from the smell which dogs were on the herbal treatment. However, the owners didn't appear to recognise the implications of the smell, as they saw only their own dog. None of the objective measurements showed any benefit of the herbal treatment over placebo, and neither did the owners' subjective assessment. However, the
investigators' subjective assessment came out as significant for the herbal at p<0.05.
The most obvious point here is that it was the measurement made subjectively by the unblinded investigators which reached significance. It's perfectly possible that even though they tried to remain impartial, once they knew which dogs were which by the smell, a little bit of wishful interpretation was going on.
The second important point however is that four different measurements were made of these dogs' condition. If you do 20 measurements comparing 2 groups in which there is actually no difference between them, one would be expected to show a significant difference at p<0.05, purely by chance. So, by doing 4 measurements, you actually have a 20% probability that one of these will show up at p<0.05, even if the groups aren't different at all.
Much of this is discussed in the paper, nevertheless the words appear, "P45FP (the herbal remedy made to sound scientific) may have a positive effect on on the clinical signs of canine osteoarthritis." Also, "P45FP appears to have a good safety profile." If I wanted to argue that P45FP was effective medicine, I'd point to this RDBPCT and quote the statistically significant measurement and these phrases. And you'd be hard put to argue with me. But in fact sensible assessment of that paper says that the only thing the herbal preparation appeared to do was make the dogs smelly.
So, here we have a very effective, licensed medicine (trilostane) with nothing in print even remotely resembling a RDBPCT, and a herbal with little or no evidence of any beneficial effect, but with a RDBPCT in print claiming statistical benefit.
You really do have to read the small print pretty carefully. But remember - anything with an obvious, self-evident effect doesn't need an RDBPCT to show efficacy, and it would probably be held to be unethical to do one. Anything which NEEDS such a study to demonstrate an effect, even if it does manage to yield a statistically significant result, probably has an effect so small that in clinical terms it's hardly worth having.
There are exceptions to this of course, particularly when looking at conditions with no effective treatment so far available, and where a new remedy might perhaps reveal a new principle which can be researched further in the hope of achieving a bigger, clinically useful effect. But in general, real drugs don't have RDBPCT evidence, and if you have to search that closely for evidence, you don't have much there at all.
Rolfe.
