Double Blind test

[Jyera]

The phrase of "Double Blind Test" should be fairly well used.

Here some questions i have.
I thought the answers to these questions would be helpful.

1. Has "Double Blind Test" been rigourously thought out as a generic methodology?

2. Has it been standardised by some world standards body such as ISO? I cannot seem to find.

3. If it is not standardised yet. Should it be standardised?

4. Is there documentation of common mistakes to avoid when implementing "Double Blind Test"?

5. Is there specific, classifiable type of situations where it could be disastrously wrong to do "Double Blind Test"?
If such classification can be identified, it would be make it easy to help educate the general public. It'll be easy to spot a case of misuse or abuse.

5. Is there any statistics about the abuse and misuse of "Double blind test"?
ie. how many organisation claimed using "double blind test" to improve creditbility of their dubious product, but actually cared less about doing it right.

 

[Dr. Imago]

The phrase of "Double Blind Test" should be fairly well used.

Here some questions i have.
I thought the answers to these questions would be helpful.

1. Has "Double Blind Test" been rigourously thought out as a generic methodology?

Yes.

2. Has it been standardised by some world standards body such as ISO? I cannot seem to find.

Standardized how?

3. If it is not standardised yet. Should it be standardised?

I'm not sure what you mean by "standardized". Perhaps you could ask the question in a different way. (I'm not being cagey; I really don't understand what information you are attempting to derive.)

4. Is there documentation of common mistakes to avoid when implementing "Double Blind Test"?

Yeah, allowing certain people involved in the trial access to the blind and/or unblinding before the data has been locked. No peeks allowed!! Conversely, not allowing access to independent monitoring committees when patient safety is an issue. These things rarely (if ever) happen anymore.

5. Is there specific, classifiable type of situations where it could be disastrously wrong to do "Double Blind Test"?
If such classification can be identified, it would be make it easy to help educate the general public. It'll be easy to spot a case of misuse or abuse.

Well, it's unethical to offer a placebo as solo therapy in situations where witholding treatment could injure a patient. For example, in cancer studies, the placebo arm is usually a masked and/or added to the standard therapy (i.e., new drug + standard therapy vs. placebo + standard therapy).

5. Is there any statistics about the abuse and misuse of "Double blind test"?
ie. how many organisation claimed using "double blind test" to improve creditbility of their dubious product, but actually cared less about doing it right.

The only "abuse" is cheating (e.g., looking at treatment arm assignments before the data is locked, inadequate randomization that allows any of the study participants to figure out which treatment they are getting, unblinding, etc.).

The only real purpose of a double-blind study is to remove bias. Randomization, most often part of double-blind studies, also helps to protect the blind and further remove selection bias.

-TT

 

[Jyera]

Standards

By Standardize I mean for the DBT (double Blind Testing methodology) to be described and documented in detail and endorsed by an international standard body.

The standard will among other things, state what must be done or achieve, to be conforming to the standard.

Standard is fairly common in various industries.
Common standards body like ISO or IETF.
Common standards like ISO 9000, ISO 14000, HTML2.0

It allow processes to be measured against the standard.

 

[Capsid]

Clinical trials must be undertaken under the standard of Good Clinical Practice

 

[Asolepius]

Re: Standards

By Standardize I mean for the DBT (double Blind Testing methodology) to be described and documented in detail and endorsed by an international standard body.

The standard will among other things, state what must be done or achieve, to be conforming to the standard.

Standard is fairly common in various industries.
Common standards body like ISO or IETF.
Common standards like ISO 9000, ISO 14000, HTML2.0

It allow processes to be measured against the standard.

The principle of randomisation and blinding is not specific to clinical trials. It is a generally accepted design for experiments across the whole range of science. There is a vast body of published literature, including many excellent textbooks, which lay down the principles and methodology. There is no formal international standard, because the regulation is carried out by the scientific community, in the form of peer-reviewed journals and congresses. That in effect is the international standard. This is not to say that no bad experiments are ever published - they are, as we see from the citations supposedly supporting homeopathy (to give just one example). Publishing is big business, and if you are not fussy about standards you can get just about anything published somewhere.

Also it's worth pointing out that, although very important, blinding is one of a number of features of a well-designed experiment.

 

[exarch]

Re: Standards

Originally posted by Jyera
By Standardize I mean for the DBT (double Blind Testing methodology) to be described and documented in detail and endorsed by an international standard body.

The standard will among other things, state what must be done or achieve, to be conforming to the standard.

Standard is fairly common in various industries.
Common standards body like ISO or IETF.
Common standards like ISO 9000, ISO 14000, HTML2.0

It allow processes to be measured against the standard.

Well, the question you raise makes sense only from a point of view of someone who doesn't quite understand what double blind testing is.
The double blind test can't be standardised any more than a highschool examination can be, because that's just what it is: a measuring device for whatever the test implementor decides (s)he wants to test.

There are certain guidelines as to what constitutes a decent, valid double blind test, and there are certain things that would invalidate the test results (like cheating for example), but when it comes right down to it, the double blind protocol can vary widely depending on the subject matter it's applied to.

The basic idea remains the same no matter where you use it though: you test something objectively by removing possible bias by the test's observer by making them unaware what significance the results have (i.e. which results correspond to which test group for instance). Usually this is achieved by testing both the item or process being tested and a similar looking dud item or process, but not telling however is collecting the test data which is which. If the test involves intelligent test subjects, you make sure of course they also don't know what's being tested on them. Since neither the test taker nor the test subjects know who got tested with what, the test is double-blinded.

 

[Iconoclast]

5. Is there specific, classifiable type of situations where it could be disastrously wrong to do "Double Blind Test"?

Sure. When testing bullet-proof vests you don't want to be one of the schmucks who has to wear a placebo.

 

[Dr. Imago]

Okay, Jyera, I think I get what you mean by "standardization" as you describe. And, I agree with what exarch said.

A test is validated when it is reproduced and gets similar (i.e., statistically significant) results. It can also be internally validated by monitoring for adherence to the study protocol (e.g., the blind was not broken during the study, the data was collected with high fidelity, the analysis of the primary endpoint was carried out precisely in the manner pre-described, etc.). So, if you ran a double blind test the correct way, you would get a meaningful result. The results would then be validated by the test. This would point to a sound methodology.

Double-blinding is a methodology. Validation and standardization is a process. Each successful double-blind test validates the methodology. You can't "standardize" a methodology, but you can demonstrate that it is effective.

Do you follow?

-TT

 

[Dr Adequate]

One problem raised recently is that the drug being tested, besides being (or not being) effective, may also have unwanted side effects. Now these effects may increase the patient's confidence in the drug ("you can feel it working") and, if the patient knows they're in a double blind trial, increases their confidence that they've actually received the drug.

So in such cases a mere placebo isn't good enough --- you want to replicate the unwanted side-effects too.

The hardest thing in science is to make sure you're measuring the thing you think you're measuring... oh, apart from the maths.

 

[Dr. Imago]

One problem raised recently is that the drug being tested, besides being (or not being) effective, may also have unwanted side effects. Now these effects may increase the patient's confidence in the drug ("you can feel it working") and, if the patient knows they're in a double blind trial, increases their confidence that they've actually received the drug.

So in such cases a mere placebo isn't good enough --- you want to replicate the unwanted side-effects too.

The hardest thing in science is to make sure you're measuring the thing you think you're measuring... oh, apart from the maths.

I've worked on trials where this was a potential factor. In one trial, we asked investigators at the end of the data collection phase for the patient to fill out a questionairre (before the blind was broken) asking whether or not they thought the patient was receiving the active compound.

The result showed was absolutely no significant correlation between what the investigators recorded and what was actually given to the patient. This further validated the study blind.

-TT

 

[Asolepius]

I've worked on trials where this was a potential factor. In one trial, we asked investigators at the end of the data collection phase for the patient to fill out a questionairre (before the blind was broken) asking whether or not they thought the patient was receiving the active compound.

The result showed was absolutely no significant correlation between what the investigators recorded and what was actually given to the patient. This further validated the study blind.

-TT

I rather like that - well done! There are of course all sorts of other reasons why maintaining a blind can be tricky, and acupuncture is a famous one. Have you heard about the sham needle which has just been introduced in Germany? Guess what - no difference in clinical trials from real needles.

 

[Jyera]

Re: Re: Standards

Agree with Exarch that it make sense only to people, who doesn't quite understand what double blind testing. I would say many of these novice are critically involved in the process.

Eg.
(1) A businessman of XYZ company declared his product Double-Blind-Tested according to ABC-DoubleBlindTesting-specification.
He is held to his promise of what he would do or had done.
(2) Novice Engineer/technician would ensure that it is tested according to what XYZ declared. The ABC-doubleBlindTesting-Spec shields him from the conflict between professional and business needs.
(3) It is easy for a Whistle-blower to specify exactly the area of non-compliance. Or the attempt to take advantage of a weak point in the ABC-DBT-Specification.
(4) The actual knowledge of a DBTesting "Expert" can be tested despite his celebrity status.
(4) A consumer knows what essential thing XYZ must have done, (And what is not done, if he knows a certain limitation in the ABC-doubleBlindTesting-specification.)
(5) Independent auditor may be send to audit XYZ against what he declared.
(6) Legal action can be taken clearly if there are dispute.
(7) Quality Control of DB-testing is, maintained.
(8) A skeptic and a believer in the product can easily agree on an independent certified Double Blind testing expert to engage. And know exactly the quality of the tester.
(9) Researchers with little knowledge of the details of actual test of two different studies can know if they have been done based on similar DBT-standard.

I think people with little knowledge of DBT plays a huge part.

 

[Dr. Imago]

Re: Re: Re: Standards

Agree with Exarch that it make sense only to people, who doesn't quite understand what double blind testing. I would say many of these novice are critically involved in the process.

Eg.
(1) A businessman of XYZ company declared his product Double-Blind-Tested according to ABC-DoubleBlindTesting-specification.
He is held to his promise of what he would do or had done.
(2) Novice Engineer/technician would ensure that it is tested according to what XYZ declared. The ABC-doubleBlindTesting-Spec shields him from the conflict between professional and business needs.
(3) It is easy for a Whistle-blower to specify exactly the area of non-compliance. Or the attempt to take advantage of a weak point in the ABC-DBT-Specification.
(4) The actual knowledge of a DBTesting "Expert" can be tested despite his celebrity status.
(4) A consumer knows what essential thing XYZ must have done, (And what is not done, if he knows a certain limitation in the ABC-doubleBlindTesting-specification.)
(5) Independent auditor may be send to audit XYZ against what he declared.
(6) Legal action can be taken clearly if there are dispute.
(7) Quality Control of DB-testing is, maintained.
(8) A skeptic and a believer in the product can easily agree on an independent certified Double Blind testing expert to engage. And know exactly the quality of the tester.
(9) Researchers with little knowledge of the details of actual test of two different studies can know if they have been done based on similar DBT-standard.

I think people with little knowledge of DBT plays a huge part.

Jyera,

No offense intended, but I am having a really hard time understanding exactly what point you're trying to make.

-TT

 

[Jyera]

Re: Re: Standards

I do not quite understand why Double blind test cannot be standardise?

Originally posted by exarch ... The double blind test can't be standardised any more than a highschool examination can be, because that's just what it is: a measuring device for whatever the test implementor decides (s)he wants to test.
...[/B]

In this specific case ...
Let say it is an English high school Exam test.
I could say that an English high school Exam test that conform to, "ABC1234: Good English Exam Standard."
Which might mean
(a) that the paper must have at least ALL the following sections, Comprehesion, Composition, Grammar, Spelling. (Not only Spelling)
(b) Paper must have 100 points of test, ie, 100 Questions (Not just 10).
(c) Paper Must be authored or edited by a teacher who has at least 2 years experience in exam paper authoring. etc ... etc ...
(d) Examination must be done in a school, not at home...

And the Education Ministry might provide incentive for all schools to conform to this standard as a min and "hygiene" level of requirement. Auditing can be done easily. And if non-conformance is correlated to falling literacy, some thing can be planned.

 

[Jyera]

Re: Re: Re: Re: Standards

Jyera,
No offense intended, but I am having a really hard time understanding exactly what point you're trying to make.
-TT

Sorry for any confusion.
Let me try to explain again...

A businessman from XYZ developed a device call ..."WineClip". which improves the taste of wine. He says that it works. And declared that it has been verified to be be truely effective. This so because they have commission their testing engineer to do a double-blind test. He also declared that they had designed their test based on the guidelines from ABC-standard-organistion for doing a proper Double-Blind-test.

Here, businessman knows nuts about double-blind testing and just wants to get as much technical jargon to push his product.

The Engineer being new, also new to double blind testing, but found comfort in getting a guideline on how to design a double-blind testing from ABC.

So far, both novice to Double-blind-test, but they are benefitting from it.

 

[jj]

The phrase of "Double Blind Test" should be fairly well used.

Here some questions i have.
I thought the answers to these questions would be helpful.

1. Has "Double Blind Test" been rigourously thought out as a generic methodology?

2. Has it been standardised by some world standards body such as ISO? I cannot seem to find.

3. If it is not standardised yet. Should it be standardised?

4. Is there documentation of common mistakes to avoid when implementing "Double Blind Test"?

5. Is there specific, classifiable type of situations where it could be disastrously wrong to do "Double Blind Test"?
If such classification can be identified, it would be make it easy to help educate the general public. It'll be easy to spot a case of misuse or abuse.

5. Is there any statistics about the abuse and misuse of "Double blind test"?
ie. how many organisation claimed using "double blind test" to improve creditbility of their dubious product, but actually cared less about doing it right.

Perhaps you should do some research into clinical psychology, audio test methodology (by the way there ARE some standards there for audio testing, look up BS1116.1 for starters), various forms of wine tasting, beer tasting, various kinds of psychometric testing, etc.

Somewhere in my library is a book on how to do DBT's for wine-tasting. I can't find it fast (it's been a while and one move since I read it), but you ought to be able to discover it without too much hardship. Regretably I don't recall the author. It wasn't much of a surprise to someone (an engineer, no less) who has lots of experience in industry-standard (and industry-leading) DBT's, so I didn't keep it at my right hand.

Rather than ask us, here, to regurgitate textbook after textbook, test after test, trial after trial, conclusion after conclusion, try looking for some information yourself. It's not REALLY that hard to find once you get into the subject.

jj - who has designed, executed, run, and been a subject for DBT's for ISO/JTC1/SC29/WG11 according to a precursor of BS1116.1 himself Well, among other things.

You might also google for BS1116.1 and MUSHRA.

 

[Jyera]

JJ

I'm not asking these questions for my personal need.
Let just say I'm facilitating answers, and accumulation of info.

I noted the forum discussed alot about using Double Blind Testing on various stuff.

But I'm curious why there isn't a thread catering to discussing or explaining Double Blind test.

And I think it would be beneficial to have a thread consolidating
info, or at least links, about Double Blind Testing procedure/process/specification. It should then be useful to all newcomer to the forum.

I found lots of link for DBT for specific field of applications.

But I did not start this thread to discuss SPECIFIC field of application for DBT. So there is no need for putting text book of information here. I didn't ask for it, and no one is compelled to answer. The last thing I'd do is to force people to answer question and make them unhappy.

But if there is opinion, saying that it is not useful to talk about DBT without specifying the specific test scenario or product, then it's fine with me that such opinion is expressed.

The answer to the questions I asked, can actually be quite simple. Yes or No. True or False. Citing reference or links to supporting evidence are most useful.

 

[Capsid]

Double blind test are difficult to standardise since as you say jyera, the design of the trial is specific to what you are testing for.

In principle, the experiment is designed to show that an effect can not be attributable to anything other than that hypothesised. No one conducting the experiment or is part of the experiment can have prior knowledge of who or what is receiving that effect. So any result observed is entirely and unequivocally objective.

I think those are the basic tenets but I have never seen this written down anywhere. It's sort of a given and what makes designing clinical trials so challenging and rewarding. My earlier post referring to good clinical practice has guidelines which state that double blind tests must be conducted but the "standard" for such tests is not provided for reasons others have already stated, namey that you can't standardise a methodology.

 

[exarch]

Originally posted by Jyera
In this specific case ...
Let say it is an English high school Exam test.
I could say that an English high school Exam test that conform to, "ABC1234: Good English Exam Standard."
Which might mean
(a) that the paper must have at least ALL the following sections, Comprehesion, Composition, Grammar, Spelling. (Not only Spelling)
(b) Paper must have 100 points of test, ie, 100 Questions (Not just 10).
(c) Paper Must be authored or edited by a teacher who has at least 2 years experience in exam paper authoring. etc ... etc ...
(d) Examination must be done in a school, not at home...

And the Education Ministry might provide incentive for all schools to conform to this standard as a min and "hygiene" level of requirement. Auditing can be done easily. And if non-conformance is correlated to falling literacy, some thing can be planned.

I repeat again, you cannot standardise testing of English like that because for example:
Schools in Belgium have 6 years primary school, 6 years secondary school, and then university. English is only taught in secondary school, and consists of 6 different levels depending on the year you're in, further more, it consists of differing levels of difficulty depending on the student's choice to follow a language oriented curriculum in 3rd year or not.

Also, it's not up to the government to teach its teachers how to compose good exams. They should have learned that when they themselves graduated. And the government already has certain standards of what kids must know at what level, but the teacher is free to choose how he wants to test his pupils for the presence or absence of that knowledge. The test is just that, a test, and it will find what it's been designed to test for by the teacher. And the teacher is the only one who really knows what he wants to find out about his students' skill-level. A boilerplate English test is going to do a poor job at assessing the students' knowledge, especially if they don't understand the wording or the point of the questions if the teacher taught them in a different way.

The same applies to double blind testing. If pharmaceutical legislation demands a manufacturer to prove efficacy, there are tests that need to be devised and implemented to test just that. How they test it is up to them, but double-blind placebo-controlled testing is the common tool for it, and that's just what they'll do. But that doesn't mean that the basic double-blind test is always the right tool for the job. It will have to be modified slightly for every occasion, and that alone would be reason enough fo the whistleblower to say: "they didn't do it exactly like this paper says they should, so they were cheating", while in fact they were simply adjusting the test to more closely test what they wanted to find out ..

And also by Jyera
A businessman from XYZ developed a device call ..."WineClip". which improves the taste of wine. He says that it works. And declared that it has been verified to be be truely effective. This so because they have commission their testing engineer to do a double-blind test. He also declared that they had designed their test based on the guidelines from ABC-standard-organistion for doing a proper Double-Blind-test.

Here, businessman knows nuts about double-blind testing and just wants to get as much technical jargon to push his product.

The Engineer being new, also new to double blind testing, but found comfort in getting a guideline on how to design a double-blind testing from ABC.

So far, both novice to Double-blind-test, but they are benefitting from it.

For starters, being able to say something was run through a double-blind test doesn't mean anything to the average consumer. What's more, you could probably say: "we did a double-blind test against placebo and didn't find anything significant." and have the consumers interpret that as:"they checked their product and nothing bad or dangerous was found."
Further more, even if there were a number of standard double-blind trial protocols they could just use as a layout for their own testing, I can see a lot more things going wrong (picking an unsuitable double-blind protocol) than if they are forced to make their own protocol and defend their reasons for doing things a certain way (why was certain data excluded, why was blinding done in such a way, etc...)..

Besides that, saying that "the wine-clip" was tested in a double-blind placebo-controlled trial already says enough to those who know what that test is. They know it was compared to a dud and any difference was recorded, although that still doesn't mean the manufacturer can't simply be lying about any positive results touted.