The Situation
My name’s Helen. I like to make cookies. So does my friend Sam. Our other friend Tim claims I make the best cookies. Yey me! However Sam and I use the same recipe and ingredients, the only difference being that Sam makes lots of small cookies while I make a few big cookies. So, Sam says Tim likes mine better because they’re bigger, not because they actually taste better.
“Not so,” I say. “Tim prefers my cookies because I’m a better cook so they taste the best!”
“Prove it!” Says Sam.
“No problem, we can design a test,” I reply. “It will have to be a double-blind test, mind…”
The Claim
“Helen’s cookies taste better than Sam’s.”
The claim doesn’t include any explanations or background; the claim simply includes the information that is to be tested.
The Test Protocol
Locations:
Required resources:
Brief test summary:
Procedure:
1. Helen and Sam each submit a copy of their cookie recipe and method.
2. Helen and Sam each make cookies in their own kitchens, starting at 2pm. An observer is present in each kitchen to ensure the previously provided recipes and methods are followed.
Steps 1 and 2 aim to ensure fairness, so neither Helen or Sam can add any extra sugar or anything!
3. When Helen states her cookies are ready, Helen’s independent observer breaks Helen’s cookies into pieces roughly 20 mm in length and breadth.
4. When Sam states his cookies are ready, Sam’s observer breaks Sam’s cookies into pieces roughly 20 mm in length and breadth.
5. Helen’s observer places 12 cookie pieces into a plastic container and closes the container.
6. Sam’s observer places 12 cookie pieces into a plastic container and closes the container.
Some steps are separately written to avoid any confusion. Breaking up the cookies should make them indistinguishable, as well as making them conveniently bite-sized!
7. At 4pm, a 3rd observer (Sophie) in the Lounge rolls a die to decide on cookie container labelling. The die result is written down by a 4th observer (James) and placed in an envelope. James leaves the sealed envelope in the Lounge and goes home.
9. At 5pm Sophie takes the cookie containers to the Lounge then goes home.
These steps are the beginning of the double-blind procedure. As Helen and Sam stay at home, no one at the Test Centre will know which box contains whose cookies after Sophie goes home. Times are specified to ensure no one enters the Lounge when they shouldn’t!
10. At 6pm a 5th observer (Lynn) goes to the Lounge and rolls the die 10 times to determine the order of each taste test. A 6th observer (Richard) also goes to the Lounge at 6pm, just to check Lynn doesn’t make a mistake.
• if 1, 2, or 3 are rolled, cookies labelled “1” will be tasted first
• if 4, 5, or 6 are rolled, cookies labelled “2” will be tasted first
The order the cookies are tasted in may affect whether Tim prefers the 1st cookie’s taste or the 2nd, so Lynn randomises the order to further ensure fairness. At this point, neither Lynn or Richard know whether Helen’s cookies are in box “1” or “2”. The test order list will be written like this:
Test 1: cookie “a” from box “1”, cookie “b” from box “2”
Etc…
Test 10: cookie “a” from box “2”, cookie “b” from box “1”
11. Also at 6pm the cookie taster (Tim) and Tim’s friend Derek go to the Bar. Tim sits in front of a table and Derek blind-folds Tim.
Tim is blind-folded to ensure the taste test results are not influenced by what the cookie pieces look like, yet an extra precaution to ensure a fair test
12. Lynn takes the test order list and two cookie containers to the Bar at 6:15pm.
13. Observed by Richard and Derek, Lynn places each piece of cookie onto a plate in the order specified by the list. Tim tastes each cookie piece and decides whether he prefers cookie “a” or cookie “b”. Lynn records Tim’s decisions.
In the Bar, neither the tester (Lynn), observers (Derek and Richard), nor the taster (Tim) know which cookies are Helen’s, so the test cannot be influenced in any way. Hence, the test is “double-blind”. 12 cookie pieces were packed into each container so that two spares are available (in case one gets dropped on the floor by accident, for example).
At this point the results should look like this:
14. Lynn compiles the results to list the number of times Tim preferred cookie “1” and the number of times Tim preferred cookie “2”, observed by Derek and Richard.
At this point the results should look like this:
Test 1: best cookie = “a” = “1”
Etc…
Test 10: best cookie = “b” = “1”
15. If the best cookie ratio “1” : “2” is equal to or greater than 7 : 3, the test result is that cookies in set “1” taste better. If the best cookie ratio “1” : “2” is equal to or less than 3 : 7, the test result is that cookies in set “2” taste better.
So if the best cookie came from box “1” 6 times, the best cookie would have come from box “2” 4 times, the ‘best cookie ratio’ would be 6:4 and the probability of Helen’s cookies tasting better than Sam’s is not significant, concluding they taste the same!
16. At 8pm, all participants return to the Bar and open the envelope (retrieved from the Lounge by James) which will reveal whether Helen or Sam’s cookies were in box “1” or “2”.
Further controls and safe-measures could be added to ensure a fair test, such as placing the sealed envelope in a locked safe, but this procedure assumes some lack of cheating intentions to reduce the number of steps described!
The test procedure should be agreed to by all participants before testing begins, ESPECIALLY step 15 as this determines the outcome of the test!
I’d like to add step 17: James buys everyone a drink to celebrate Helen’s fantastic cooking skills, as the test will have a 9:1 result in her cookies’ favour
My name’s Helen. I like to make cookies. So does my friend Sam. Our other friend Tim claims I make the best cookies. Yey me! However Sam and I use the same recipe and ingredients, the only difference being that Sam makes lots of small cookies while I make a few big cookies. So, Sam says Tim likes mine better because they’re bigger, not because they actually taste better.
“Not so,” I say. “Tim prefers my cookies because I’m a better cook so they taste the best!”
“Prove it!” Says Sam.
“No problem, we can design a test,” I reply. “It will have to be a double-blind test, mind…”
The Claim
“Helen’s cookies taste better than Sam’s.”
The claim doesn’t include any explanations or background; the claim simply includes the information that is to be tested.
The Test Protocol
Locations:
• Helen’s kitchen
• Sam’s kitchen
• Test centre, the local pub, that has two separately enclosed rooms, the Lounge and the Bar.
Required resources:
• Helen and Sam provide their own ingredients
• Two identical plastic containers with lids
• 7 observers/helpers
• paper, pens, envelope
Brief test summary:
• 2pm: Helen and Sam make cookies
• 4pm: Helen and Sam’s cookies are labelled “1” or “2” by Sophie
• Lynn lists the tasting order for 10 tests (whether “1” or “2” is tasted first)
• 6:15pm: Lynn takes the cookies to the blind-folded Tim in the Bar
• Tim does 10 taste tests, deciding whether he prefers cookie “a” or cookie “b”
• Lynn records the results, checking whether cookies “1” or “2” are favourites
• 8pm: James comes to the Bar and buys everyone a drink!
Procedure:
1. Helen and Sam each submit a copy of their cookie recipe and method.
2. Helen and Sam each make cookies in their own kitchens, starting at 2pm. An observer is present in each kitchen to ensure the previously provided recipes and methods are followed.
Steps 1 and 2 aim to ensure fairness, so neither Helen or Sam can add any extra sugar or anything!
3. When Helen states her cookies are ready, Helen’s independent observer breaks Helen’s cookies into pieces roughly 20 mm in length and breadth.
4. When Sam states his cookies are ready, Sam’s observer breaks Sam’s cookies into pieces roughly 20 mm in length and breadth.
5. Helen’s observer places 12 cookie pieces into a plastic container and closes the container.
6. Sam’s observer places 12 cookie pieces into a plastic container and closes the container.
Some steps are separately written to avoid any confusion. Breaking up the cookies should make them indistinguishable, as well as making them conveniently bite-sized!
7. At 4pm, a 3rd observer (Sophie) in the Lounge rolls a die to decide on cookie container labelling. The die result is written down by a 4th observer (James) and placed in an envelope. James leaves the sealed envelope in the Lounge and goes home.
• if 1, 2, or 3 are rolled, Helen’s cookie container will be labelled “1” and Sam’s cookie container will be labelled “2”
• if 4, 5, or 6 are rolled, Sam’s cookie container will be labelled “1” and Helen’s cookie container will be labelled “2”
8. Sophie drives round to Helen’s house to label and pick up her cookie container, then Sophie drives round to Sam’s house to label and pick up his cookie container. Helen and Sam stay at home.9. At 5pm Sophie takes the cookie containers to the Lounge then goes home.
These steps are the beginning of the double-blind procedure. As Helen and Sam stay at home, no one at the Test Centre will know which box contains whose cookies after Sophie goes home. Times are specified to ensure no one enters the Lounge when they shouldn’t!
10. At 6pm a 5th observer (Lynn) goes to the Lounge and rolls the die 10 times to determine the order of each taste test. A 6th observer (Richard) also goes to the Lounge at 6pm, just to check Lynn doesn’t make a mistake.
• if 1, 2, or 3 are rolled, cookies labelled “1” will be tasted first
• if 4, 5, or 6 are rolled, cookies labelled “2” will be tasted first
The order the cookies are tasted in may affect whether Tim prefers the 1st cookie’s taste or the 2nd, so Lynn randomises the order to further ensure fairness. At this point, neither Lynn or Richard know whether Helen’s cookies are in box “1” or “2”. The test order list will be written like this:
Test 1: cookie “a” from box “1”, cookie “b” from box “2”
Etc…
Test 10: cookie “a” from box “2”, cookie “b” from box “1”
11. Also at 6pm the cookie taster (Tim) and Tim’s friend Derek go to the Bar. Tim sits in front of a table and Derek blind-folds Tim.
Tim is blind-folded to ensure the taste test results are not influenced by what the cookie pieces look like, yet an extra precaution to ensure a fair test
12. Lynn takes the test order list and two cookie containers to the Bar at 6:15pm.
13. Observed by Richard and Derek, Lynn places each piece of cookie onto a plate in the order specified by the list. Tim tastes each cookie piece and decides whether he prefers cookie “a” or cookie “b”. Lynn records Tim’s decisions.
In the Bar, neither the tester (Lynn), observers (Derek and Richard), nor the taster (Tim) know which cookies are Helen’s, so the test cannot be influenced in any way. Hence, the test is “double-blind”. 12 cookie pieces were packed into each container so that two spares are available (in case one gets dropped on the floor by accident, for example).
At this point the results should look like this:
Test 1: cookie “a” from box “1”, cookie “b” from box “2”, best cookie = “a”
Etc…
Test 10: cookie “a” from box “2”, cookie “b” from box “1”, best cookie = “b”
14. Lynn compiles the results to list the number of times Tim preferred cookie “1” and the number of times Tim preferred cookie “2”, observed by Derek and Richard.
At this point the results should look like this:
Test 1: best cookie = “a” = “1”
Etc…
Test 10: best cookie = “b” = “1”
15. If the best cookie ratio “1” : “2” is equal to or greater than 7 : 3, the test result is that cookies in set “1” taste better. If the best cookie ratio “1” : “2” is equal to or less than 3 : 7, the test result is that cookies in set “2” taste better.
So if the best cookie came from box “1” 6 times, the best cookie would have come from box “2” 4 times, the ‘best cookie ratio’ would be 6:4 and the probability of Helen’s cookies tasting better than Sam’s is not significant, concluding they taste the same!
16. At 8pm, all participants return to the Bar and open the envelope (retrieved from the Lounge by James) which will reveal whether Helen or Sam’s cookies were in box “1” or “2”.
Further controls and safe-measures could be added to ensure a fair test, such as placing the sealed envelope in a locked safe, but this procedure assumes some lack of cheating intentions to reduce the number of steps described!
The test procedure should be agreed to by all participants before testing begins, ESPECIALLY step 15 as this determines the outcome of the test!
I’d like to add step 17: James buys everyone a drink to celebrate Helen’s fantastic cooking skills, as the test will have a 9:1 result in her cookies’ favour