Designing a test for the magnetic coasters...

[drkitten]

After the tasters have done their tasting, they answer the following multiple choice question.

"Comparing your glass of wine 1 to glass of wine 2...."
"a) You could not tell any difference between the two."
"b) Glass 1 was better than glass 2"
"c) Glass 2 was better than glass 3"

Then compare the tasters' answers to the dispersing records. Do the results line up better than would be expected by chance? Even if they consistently choose the magnetically treated wine as inferior, that would be a surprising result. It would indicate that the magnet at least did something. Or, even consider that one taster would pick the magnetically treated wine consistantly better, and another picked it as worse, that would be worth further study as well.

Then, it would probably be a good idea to rerun the test in the same manner, but eliminate choice "a" (could tell any difference) and see if forcing them to pick one or the other gives any better results.

.

What would be the advantage of this test over the triangle test as proposed earlier? As far as I can tell, it will simply make the analysis more difficult (you need to break results down by each subject), increase the amount of data required (each subject needs to drink enough to make analysis of their individual results meaningful), and reduce the overall power of the statistics.

I think it's silly to check for tasting "better" if there's no evidence even to support "different."

 

[Mercutio]

Perhaps the taste test should include a round in which everyone knows which sample they are tasting, similar to the protocol in which dowsers first demonstrate their technique on open containers of water, to prevent claims that other factors "interfered" with the magnets...

Agreed. In Randi's workshop at TAM2, they emphasized this part of the challenge.

 

[SGT]

I don't believe that either of these were
actually Coghill's claims. I think we owe
him the courtesy of testing him on the claims
he made, and not necessarily on the claims
made by some other nutcase who makes
a similar product.

You have a good point. For Coghill's coasters we should limit the test to water and wine.
For water, it should be chosen a tasteless, low mineralized one. The french Volvic would be well suited. I don't know if it is available in London, but there should be others in the market that fill the requirements.
For the wine, we should discard both cheap and very good products.
Nothing can improve a cheap wine. For a very good wine, a small improvement would not be detectable. So we shoul focus in medium quality, low bodied wines, like Valpolicella or Beaujolais Villages.
Two non magnetized coasters of the same dimensions of the products under test should be provided. The four coasters should be put into opaque numbered envelopes.
A person that does not know what is in each envelope chooses two of them. We have 50% of chance of a 'real' and a fake coaster and 25% each of two reals or two fakes.
Liquid from the same bottle is to be poured in two jugs and each jug put over one coaster. After one hour liquid of each jug is put in two separate glasses for each tester. Do not use plastic goblets, wich have smell that could mask the taste and aroma of the beverage.
Each tester records his/her impressions over the existent or inexistent differences in the two glasses.
After that, another set of two coasters is to be chosen and the experiment repeated.
Of course, the inhability to detect differences could be blamed in the poor senses of taste and smell of the testers, so the ideal test should have people from Coghill or at least some believers among the testers.

 

[drkitten]

A person that does not know what is in each envelope chooses two of them. We have 50% of chance of a 'real' and a fake coaster and 25% each of two reals or two fakes.
Liquid from the same bottle is to be poured in two jugs and each jug put over one coaster. After one hour liquid of each jug is put in two separate glasses for each tester. Do not use plastic goblets, wich have smell that could mask the taste and aroma of the beverage.
Each tester records his/her impressions over the existent or inexistent differences in the two glasses.

What advantage does this have over the much simpler and more powerful triangle test?

 

[SGT]

What advantage does this have over the much simpler and more powerful triangle test?

If people find differences from beverages that have both been submited to the magnetic treatment (or both not submited), we can suppose that all findings are subjective.
If people know that one of the coasters is 'real' and the other not, they may be influenced in finding differences in taste. Even if they think the liquid from the fake coaster tastes better, this can be attributed to personal likings.

 

[scotth]

What would be the advantage of this test over the triangle test as proposed earlier? As far as I can tell, it will simply make the analysis more difficult (you need to break results down by each subject), increase the amount of data required (each subject needs to drink enough to make analysis of their individual results meaningful), and reduce the overall power of the statistics.

I think it's silly to check for tasting "better" if there's no evidence even to support "different."

No real advantage. The analysis wouldn't be really any more difficult. You should still break it down for each subject, the possibility that one subject would perform better than others would be of interest regardless of testing method.

The test also aligns well with the actual claims. The claims say that it will make water and wine taste better. I still agree with you, that there is no evidence to support even "different".

However, a triangle test would work just as well.

Controlling for possible top pour, middle pour, bottom pour bias gets a little more difficult when splitting the bottle three ways. It also requires 3 identical decanters rather than just 2.

 

[drkitten]

If people find differences from beverages that have both been submited to the magnetic treatment (or both not submited), we can suppose that all findings are subjective.
If people know that one of the coasters is 'real' and the other not, they may be influenced in finding differences in taste.

I don't think I understand you. Could you be more explicit, please?

One of the advantages of the triangle test is that it's a forced-choice test, so people will be "compelled" to identify subtle differences in taste. That's an advantage in this context, since it prevents shilly-shallying. But the structure of the test controls for nonexistent differences by comparison to a 1/3 baseline accuracy. If they find differences in taste that aren't there, then they'll randomly select a "different" sample and the statistics will filter it out.

 

[drkitten]

Perhaps the taste test should include a round in which everyone knows which sample they are tasting, similar to the protocol in which dowsers first demonstrate their technique on open containers of water, to prevent claims that other factors "interfered" with the magnets...

I don't think that this is appropriate in this instance. In the event that there is, in fact, no discernable effect, then an honest panel of tasters will report that on the open test. More likely, a gathering of skeptics would specifically report "no discernable difference," even in the event that there was (confirmation bias works both ways). In the case of the Randi tests, you can usually count on a participant being sufficiently secure in his own powers to agree that his powers work in an open test; that assumption doesn't hold here.

Furthermore, this is a commercial product. If it doesn't, in fact, have any effect in controlled testing, then issues of "interference" are at least partially irrelevant. As long as there are no contraindications in the instructions, a commercial product should work as advertised.

 

[patnray]

Furthermore, this is a commercial product. If it doesn't, in fact, have any effect in controlled testing, then issues of "interference" are at least partially irrelevant. As long as there are no contraindications in the instructions, a commercial product should work as advertised.

Agreed, it should. But advertisers/promoters have no obligation to point out the shortcomings of their products. Recall that the sellers of bogus audiophile products, such as cable that cost 10 times as much as ordinary coopper wire, when confronted with tests that prove people cannot hear any difference using their product and ordinary products, resort to claims that the listeners do not have sensitive enough ears to hear the difference. But do they tell you, in any of their sales material, that they believe only 1 in 100 people can hear the difference?

The point is to eliminate that kind of wiggle room.

 

[SGT]

I don't think I understand you. Could you be more explicit, please?

One of the advantages of the triangle test is that it's a forced-choice test, so people will be "compelled" to identify subtle differences in taste. That's an advantage in this context, since it prevents shilly-shallying. But the structure of the test controls for nonexistent differences by comparison to a 1/3 baseline accuracy. If they find differences in taste that aren't there, then they'll randomly select a "different" sample and the statistics will filter it out.

As you said in your answer to patnray , skeptics would be biased and report no difference in taste with two or three samples. We should limit the panel of testers to believers in the power of magnets, homeopathy etc.
If these people know that one of the recipients contains the magnetized liquid, they will naturally choose one of them, even if they don't feel any difference.
In the test I proposed they don't know if there is one, two or no magnetized liquids, so they possibly will more honestly find no difference.

 

[roger]

As you said in your answer to patnray , skeptics would be biased and report no difference in taste with two or three samples. We should limit the panel of testers to believers in the power of magnets, homeopathy etc.

Testee shouldn't what is being tested. Tell 'em your testing different waters. Or different mineral additives in the water. Or something.

 

[Soapy Sam]

This should involve people who have no idea of what is being tested.
There ought to be double blind protocols.
It should be done with bottled water- perhaps several brands.
(Making experimental subjects drink London tap water would be unethical).
Don't use wine- some folk like cheap wines. (I'm one). Tastes differ.
There should be a dummy disc - non magnetic- , possibly several, to confuse the issue and all the discs should be hidden from testees.
In fact better they don't know the discs even exist.
Testees should not drink between tests- a pub is emphatically the wrong place for this. Your results would not be taken seriously , no matter what they showed.

Whoever records the data should have no idea what disc is where.

This may all be obvious. If so, sorry.
I will be happy to help out with any costs you incur. Sorry, I live 400 miles from London and I'm a great deal farther right now.
Have you tried Athon?

 

[scotth]

This should involve people who have no idea of what is being tested.
There ought to be double blind protocols.
It should be done with bottled water- perhaps several brands.
(Making experimental subjects drink London tap water would be unethical).
Don't use wine- some folk like cheap wines. (I'm one). Tastes differ.
There should be a dummy disc - non magnetic- , possibly several, to confuse the issue and all the discs should be hidden from testees.
In fact better they don't know the discs even exist.
Testees should not drink between tests- a pub is emphatically the wrong place for this. Your results would not be taken seriously , no matter what they showed.

Whoever records the data should have no idea what disc is where.

This may all be obvious. If so, sorry.
I will be happy to help out with any costs you incur. Sorry, I live 400 miles from London and I'm a great deal farther right now.
Have you tried Athon?

Pretty good points, but it kinda takes some of the fun out of it. You might want a believer to witness/participate in the running of your experiment. Make sure you can't be called for cheating.

 

[richardm]

Controlling for possible top pour, middle pour, bottom pour bias gets a little more difficult when splitting the bottle three ways. It also requires 3 identical decanters rather than just 2.

You could always buy three bottles, mix the contents in a pan and then redecant back into the bottles. It doesn't really matter what happens to the wine, since it's whether the magic coaster changes the taste that is the important thing.

 

[scotth]

You could always buy three bottles, mix the contents in a pan and then redecant back into the bottles. It doesn't really matter what happens to the wine, since it's whether the magic coaster changes the taste that is the important thing.

That sounds workable.

 

[EHocking]

I don't believe that either of these were
actually Coghill's claims. I think we owe
him the courtesy of testing him on the claims
he made, and not necessarily on the claims
made by some other nutcase who makes
a similar product.

A further claim that they make is for the Pet Coaster.
<http://www.galonja.co.uk/galonja_sh...&g_u_nam=&g_tim=&pid=92&v_det=1&full=1&c_id=0>
"Given the choice your pet will always choose to drink magnetic water, they can tell the difference. Magnetic water is more natural. Using a pet coaster ensures that your pet receives maximum benefit from their drinking water. They will love the taste. "

Here at least you have test subjects that are won't even know that they are being tested.

Off the top of my head, you could set up two identical bowls, one with a coaster in it and one without, behind "cat flaps". You could then fix a counter to the flap and monitor the count daily.

You'd probably want to change the location of the magnets to account in case things like proximity to doors or hours in sunlight, or just your pets habit may affect the numbers.

As a further control, I'd run the same experiment with NO magnets in either dish and correlate the numbers against the numbers where magnets were present.

Note that there is a simple failure case. The claim is that

"your pet will *always* choose to drink magnetic water"

So statistical significance is practically ruled out - it's 100% or a failure.

Oh, and I'm London based if there is Red Ned in the offing.

 

[drkitten]

As you said in your answer to patnray , skeptics would be biased and report no difference in taste with two or three samples.

Not an option in a forced-choice situation.

 

[scotth]

Not an option in a forced-choice situation.

Sure it is, just report the the odd man out without regard to taste.

 

[T'ai Chi]

Re: Re: Designing a test for the magnetic coasters...

I recommend a "triangle test" as a test methodology. Get three identical bottles of water, chill them identically, take the out of the fridge, open them, and "magnetize" one. Pour three samples into three identical labelled glasses (cheap plastic is probably sufficient), then have the subjects determine which of the three "tastes different."

Probably a good idea to have water biscuits or something available as a palate cleanser.

Obviously, you need to do the pouring and labelling in a different area from where the people are being served, and you want a confederate to serve and tabulate the results.

Expected results are that 1/3 (0.3333) of the participants would correctly identify the magnetized water by chance; anything "significantly" better than 1/3 is evidence that Coghill is onto something. I can crunch some numbers quickly if you want cookbook stats.

This is basically identical to a test I set up and supervised as a teaching assistant in grad school with 3 different sources of water (2 were expensive bottled waters, 1 was just water from the tap down).

Yeah, it was just a test of proportions, testing if p = 1/3 or if p is different from 1/3.

 

[T'ai Chi]

For the taste test, I recommend using wine.

I'd personally use water, given that after drinking wine ones judgements can be distorted.