Bias in Challenge Protocol?

[ProbeX]

I'm all for the objective testing of alleged paranormal activity and abilities. But:

Doesn't the fact that claimants aren't allowed to collaborate on the choice of judges, experts and intermediaries, etc., create too much potential for bias behavior and unchecked duplicity from Randi's organization and/or his appointed experts?

Also, in keeping w standard scientific protocol, observing parties in a test trial aren't supposed to be invested in a particular outcome. But doesn't the fact that Randi is risking the loss of his appointed organizational funds (and possibly, his reputation), place him in a position of non-neutrality and bias in scientifically or objectively determining what or who (de facto) is or isn't paranormal?

Are the people he hires or uses for final testing trials, by extension, suspect of the same potential bias? ... Why isn't there a mutual agreement on who helps facilitate the final round of testing?

 

[GzuzKryzt]

I'm all for the objective testing of alleged paranormal activity and abilities. But:

Doesn't the fact that claimants aren't allowed to collaborate on the choice of judges, experts and intermediaries, etc., create too much potential for bias behavior and unchecked duplicity from Randi's organization and/or his appointed experts?

Also, in keeping w standard scientific protocol, observing parties in a test trial aren't supposed to be invested in a particular outcome. But doesn't the fact that Randi is risking the loss of his appointed organizational funds (and possibly, his reputation), place him in a position of non-neutrality and bias in scientifically or objectively determining what or who (de facto) is or isn't paranormal?

Are the people he hires or uses for final testing trials, by extension, suspect of the same potential bias? ... Why isn't there a mutual agreement on who helps facilitate the final round of testing?

There is a mutual agreement, ProbeX: It is called the "Protocol".

Strict adherence to the mutually agreed upon Challenge Protocol neither involves judging, judgment, judges nor potential bias. Hence, it is irrelevant who facilitates the test, since the results are self-evident.

Have you read the Challenge Application section? Preliminary tests happened with Achau Nguyen, Angela Patel and Hans Peter Borer. Do those protocols include the potential bias you referred to?

 

[ProbeX]

For example Gzuz, here is a quote from Kramer, detailing a part of the procedure that took place in the case of Achau Nguyen:

"So, this reflects an all too common hardship JREF encounters when attempting to enlist qualified investigators to help with claims. Very often we can eventually locate someone we feel is qualified (in this particular case NOT an experienced paranormal phenomenon investigator but a scientist and critical thinker, which will do just fine ..."

Who qualifies is decided solely by Randi's "camp", if I'm not mistaken. But why don't both parties have a say in who gets to observe and participate in the testing process? Where there's money to be gained or lost, neither party can be taken on their word or considered to have neutral involvement or benevolent motives in choice of study participants. From what I've read here, the preliminary test includes negotiation. Great. But this doesn't appear to be the case in the final test protocol. ... Or show me where the protocol explicitly states something to the contrary.

The main question I have is:

What justifies the granting of unilateral discretion in making choices about experts' participation, in an adversarial challenge where money is at stake for one party and up for grabs for the other - regardless of whether it's a scientist or a paranormal "expert" who's chosen?

 

[GzuzKryzt]

For example Gzuz, here is a quote from Kramer, detailing a part of the procedure that took place in the case of Achau Nguyen:

"So, this reflects an all too common hardship JREF encounters when attempting to enlist qualified investigators to help with claims. Very often we can eventually locate someone we feel is qualified (in this particular case NOT an experienced paranormal phenomenon investigator but a scientist and critical thinker, which will do just fine ..."

Who qualifies is decided solely by Randi's "camp", if I'm not mistaken. But why don't both parties have a say in who gets to observe and participate in the testing process? Where there's money to be gained or lost, neither party can be taken on their word or considered to have neutral involvement or benevolent motives in choice of study participants. From what I've read here, the preliminary test includes negotiation. Great. But this doesn't appear to be the case in the final test protocol. ... Or show me where the protocol explicitly states something to the contrary.

The main question I have is:

What justifies the granting of unilateral discretion in choice of expert participation, where money is at stake (regardless of whether it's a scientist or a paranormal "expert" who's chosen)?

The justification is simple: It is the JREF Challenge.

Impartial behaviour is easily verifiable, because the tests are videotaped. Any test is reproducable by any person with any "referee", based on the given protocol.



The preliminary test uses the same protocol as the final or formal test. http://www.randi.org/research/faq.html#5.2

 

[ProbeX]

The justification is simple: It is the JREF Challenge.

Impartial behaviour is easily verifiable, because the tests are videotaped. Any test is reproducable by any person with any "referee", based on the given protocol.

Video documentation is great but doesn't ensure the testing methodology used is set about in a non-bias way that includes the consultation of non-partisan experts from both camps.

The design may have hidden flaws because one side (w a vested interest in holding onto money and upholding a opposing paradigm) has the final say in the design. Randi is a magician, not a scientist with a Ph.D, as far as I know. And if he hires Ph.Ds, again, we run into the problem of design potentially biased by vested interest in the outcome he expects.

Sure it's the JREF Challenge and Randi is free to do whatever he pleases, including using biases in methodology.

But such a unilateral approach doesn't ensure a scientifically unbias and respectable research design. Also, the protocol appears to be somewhat shaped by both parties, which is good. But again, the chosen experts should be taken from a neutral pool, by lottery, for example.

There should also be mutually agreeable observers from both sides present, since the person offering his blessing to the test conditions is a magician and may be emploring others to create an illusion of non-evidence (which could produce video "evidence", instead of evidence lol), driven by a desire to hold on to a reputation and the money belonging to the organization he represents.

The chances of his behaving this way may seem remote to an avid Randi fan, but it doesn't lend enough security to the applicant. You have to implore logic, w/out giving either side an unfair edge in design, review and "verdict", if you wish to be fair. But of course he's free to toss the gauntlet down in a careless manner if he desires since he oversees the "challenge" (which sounds quasi-scientific, frankly).

 

[gnome]

It's not unilateral though... the entire test procedure, including its participants, must be agreed upon by both parties, or it doesn't happen. I can't imagine any other alternative that doesn't force one side or the other to participate in a test they don't agree with.

 

[GzuzKryzt]

...
The design may have hidden flaws because one side (w a vested interest in holding onto money and upholding a opposing paradigm) has the final say in the design.

Where has this happened in the past?

Sure it's the JREF Challenge and Randi is free to do whatever he pleases, including using biases in methodology.

Again, where has this happened in the past?

But such a unilateral approach doesn't ensure a scientifically unbias and respectable research design. Also, the protocol appears to be somewhat shaped by both parties, which is good. But again, the chosen experts should be taken from a neutral pool, by lottery, for example.
...

The protocol negotiations, etc. were open to the public in the past. JREF actions stand not only under the scrutiny of their opponents, but also of their donors, beneficiaries, etc.

JREF simply can't allow bias. It would result in a negative feedback; from people applying common sense, that is.

 

[GzuzKryzt]

...
But of course he's free to toss the gauntlet down in a careless manner if he desires since he oversees the "challenge" (which sounds quasi-scientific, frankly).

ProbeX, please read some Challenge Applications http://www.internationalskeptics.co...?f=43&daysprune=-1&order=desc&sort=replycount
which did progress to the protocol negotiation stage, and please point out specific examples where you consider JREF's behaviour unnecessarily careless, quasi-scientific, biased, etc.

 

[ChaoticLimbs]

A completely equal share in this would be fair IF the applicant agreed to pay the JREF one million dollars should he/she fail to perform as claimed, so that both have something to lose and something to gain.

Otherwise, why should the JREF not be picky as to whose word they will accept? The reason is not one of doubt, but one of the knowledge of the methods of fraud and deception.

 

[Cuddles]

Just because most applicants don't seem to have any friends to help with the test does not mean they are not allowed. Of course it would be best if there are observers present from both sides, but for some reason reason the applicants themselves never seem to worry about this. In any case, it doesn't really matter. The tests are designed so that no interpretation is required, a pass will be self evident. For example, for someone claiming to influence a coin toss, if they get 10 heads in a row it doesn't matter who is looking at them, they are still 10 heads. Since the applicant themselves can see the result they can act as their own observer.

In the end it comes down to the applicant. The JREF appoints a representative and approves the protocol. Any extra people or resources are entirely up to the applicant. If they don't bother asking anyone else to help that is their own problem and not the fault of the JREF.

 

[Moochie]

I'm all for the objective testing of alleged paranormal activity and abilities. But:

Doesn't the fact that claimants aren't allowed to collaborate on the choice of judges, experts and intermediaries, etc., create too much potential for bias behavior and unchecked duplicity from Randi's organization and/or his appointed experts?

Also, in keeping w standard scientific protocol, observing parties in a test trial aren't supposed to be invested in a particular outcome. But doesn't the fact that Randi is risking the loss of his appointed organizational funds (and possibly, his reputation), place him in a position of non-neutrality and bias in scientifically or objectively determining what or who (de facto) is or isn't paranormal?

Are the people he hires or uses for final testing trials, by extension, suspect of the same potential bias? ... Why isn't there a mutual agreement on who helps facilitate the final round of testing?

Here we go again...

M.

 

[ProbeX]

Where has this [design flaws & methodology] happened in the past?

Problem is there's no way of knowing when or if this happened. Again, he's a magician. I can't blindly accept that he or one or his personally chosen watchdogs or expert observers wouldn't tamper w/video evidence, etc.

There need to be representative experts and "watchdogs" chosen and agreed upon by both camps during all parts of the process to best avoid the possibility. Ditto for the question of proper methodology. If it were me I'd have a lawyer by my side through all negotiations and during all tests. Randi should not be "dropping in" on any part of the test procedure either. Anyone w a vested interest (pot. loss of money is a qualifier) should not come into contact w any of the test subjects. This particular piece of wise scientific protocol is ignored by Dr. Gary Schwart (Human Labs) as well.

I am not debating the fact that there's a degree of negotiation from both sides granted in the terms listed at this site (so you can refrain from reposting the link lol). I just think it should extend to include a joint choice of experts and clinicians ... and there should be a third party to arbitrate.

The final judge/s of evidence (it could literally be a court room judge), would be jointly chosen by way of lottery, to ensure this happened.

Someone w a vested interest in the outcome (Randi) should not have any part in judging the final outcome, or have the final nod re: whether or not proof was demonstrated. Someone might say, well his money (or rather, the foundations' money lol) allows him to do this. Well his money may allow him to, but his choice to do so negates the scientifically and ethical viability of the test/s.

What I'm suggesting is reasonable and expected, from the perspective of classic scientific methodology.

 

[Terry]

Someone w a vested interest in the outcome (Randi) should not have any part in judging the final outcome

which is why the protocol is always set up so that no judgement is required, only counting and comparing to a previously agreed (by both parties) number.

 

[Moochie]

Problem is there's no way of knowing when or if this happened. Again, he's a magician. I can't blindly accept that he or one or his personally chosen watchdogs or expert observers wouldn't tamper w/video evidence, etc.

There need to be representative experts and "watchdogs" chosen and agreed upon by both camps during all parts of the process to best avoid the possibility. Ditto for the question of proper methodology. If it were me I'd have a lawyer by my side through all negotiations and during all tests. Randi should not be "dropping in" on any part of the test procedure either. Anyone w a vested interest (pot. loss of money is a qualifier) should not come into contact w any of the test subjects. This particular piece of wise scientific protocol is ignored by Dr. Gary Schwart (Human Labs) as well.

I am not debating the fact that there's a degree of negotiation from both sides granted in the terms listed at this site (so you can refrain from reposting the link lol). I just think it should extend to include a joint choice of experts and clinicians ... and there should be a third party to arbitrate.

The final judge/s of evidence (it could literally be a court room judge), would be jointly chosen by way of lottery, to ensure this happened.

Someone w a vested interest in the outcome (Randi) should not have any part in judging the final outcome, or have the final nod re: whether or not proof was demonstrated. Someone might say, well his money (or rather, the foundations' money lol) allows him to do this. Well his money may allow him to, but his choice to do so negates the scientifically and ethical viability of the test/s.

What I'm suggesting is reasonable and expected, from the perspective of classic scientific methodology.

It is clear to me that you haven't bothered to acquaint yourself with all the information regarding the MDC on this web site. Why don't you do that before boring us with your inane assumptions?

And, by the way, the Challenge has nothing to do with science, as you will discover when you read through the site.

M.

 

[GzuzKryzt]

Problem is there's no way of knowing when or if this happened. Again, he's a magician. I can't blindly accept that he or one or his personally chosen watchdogs or expert observers wouldn't tamper w/video evidence, etc.

There need to be representative experts and "watchdogs" chosen and agreed upon by both camps during all parts of the process to best avoid the possibility. Ditto for the question of proper methodology. If it were me I'd have a lawyer by my side through all negotiations and during all tests. Randi should not be "dropping in" on any part of the test procedure either. Anyone w a vested interest (pot. loss of money is a qualifier) should not come into contact w any of the test subjects. This particular piece of wise scientific protocol is ignored by Dr. Gary Schwart (Human Labs) as well.

I am not debating the fact that there's a degree of negotiation from both sides granted in the terms listed at this site (so you can refrain from reposting the link lol). I just think it should extend to include a joint choice of experts and clinicians ... and there should be a third party to arbitrate.

The final judge/s of evidence (it could literally be a court room judge), would be jointly chosen by way of lottery, to ensure this happened.

Someone w a vested interest in the outcome (Randi) should not have any part in judging the final outcome, or have the final nod re: whether or not proof was demonstrated. Someone might say, well his money (or rather, the foundations' money lol) allows him to do this. Well his money may allow him to, but his choice to do so negates the scientifically and ethical viability of the test/s.

What I'm suggesting is reasonable and expected, from the perspective of classic scientific methodology.

ProbeX, at the risk of sounding redundant: A proper, mutually agreed-upon protocol does not involve judgment on the ensuing test results. They are self-evident.

I refer you to my protocol proposal for PeaceCrusader test:

...
A brief protocol proposal:

1. Person A prepares 10 boxes with 10 different items, the boxes are numbered 01 - 10. Person A notes on a sheet of paper what item is in what box. Person A seals the boxes and deposits them on a table in a room. No one else is present in said room. The table is monitored with a video camera. Person A must not have contact with anyone else involved in this test before the conclusion of step #5.
2. Person B enters the room accompanied by the "medium".
3. The "spirit" "contacts" the "medium" with the information about the content of the boxes. A time limit for this part of the test is set, e.g. two hours = 12 minutes on average per box. The "medium" is not allowed to touch the boxes.
4. Person B notes what information the "spirit" gave about which particular box.
5. The "medium" will tell Person B when the "contact" is finished. If the test duration exceeds the pre-agreed time limit, Person B will declare the test finished.
6. After that, Person A will reenter the room and both lists will be checked for matches. Six or more matches will mean a successful test. Five and less matches will mean an unsuccessful test.

There are a couple of things which need to be addressed first:

a. Who will apply for the Challenge? Only the test participant is allowed to apply, as stated here: http://www.randi.org/research/challenge.html
Since this seems to be the "medium", this means she would have to apply herself.
b. The protocol is mutually agreed upon between JREF and the applicant. I am neither a representative of JREF, nor are you, PeaceCrusader, the one who will apply.
c. Only after a proper application is received and accepted, JREF and the applicant can negotiate a test location, a test date and the protocol.
d. There's probably more details to work out. For the sake of brevity I only addressed the most important points.
...

Referring to your first paragraph: Do you have any evidence of said "tampering" with video evidence? What do you mean by "Again, he's a magician."?

If you took the time and read through the Challenge Applications of Ian Conger, Achau Nguyen, Carina Landin, Prophet Yahweh, Angela Patel, Hans Peter Borer and Beth Clarkson, you will probably have a different perspective on how JREF behaves towards their applicants.

I agree, it takes quite some time and patience. Are you up to it?

 

[T'ai Chi]

which is why the protocol is always set up so that no judgement is required, only counting and comparing to a previously agreed (by both parties) number.

Who judges that the claim is paranormal in nature to begin with?

Doesn't that imply that the judger knows about all powers in the universe if he knows what is paranormal and what is not before science has pronounced it so?

 

[paradigm]

In all honesty there seems to be a little bit of bias. Often it seems as if the testing organization has the attitude that "Oh this person isn't going to make it anyway, why bother?" Naturally this is somewhat destructive and can lead to mistakes in protocol implementation. Furthermore from the standpoint of the challenger they are essentially acting in desperation and encouraged to tolerate any irregularities. Despite what is said to them to the contrary. After all, if they deny the challenge as given, do they know they will be gven another opportunity? I have seen Randi himself advise that if the challenger does not feel the test is right, do not administer the test. Some challengers may take this to mean "take it or leave it" rather than "we can fix the mistakes or problems you have and then try again later". It is much like a person whom desperately needs a loan. But the only bank in town presents a 30% APR. Well then it is either take it or leave it and the challenger almost surely gets this sense. Don't get me wrong now. For the most part Randi and the skeptic organizations do a great job. Especially considering how hard of a job it is. But there is the unavoidable human factor in all this which should not be denied.

 

[Terry]

Who judges that the claim is paranormal in nature to begin with?

Doesn't that imply that the judger knows about all powers in the universe if he knows what is paranormal and what is not before science has pronounced it so?

As I understand it (and I do not speak for the JREF), that is Randi's judgement call. For instance, if someone claimed to have a telepathic link with another person, and a protocol was agreed, and they passed, they'd win the million. If it was later discovered using fMRI that the person had an extra organ in their brain that gave them this power, and after more study, it was understood how it worked, then fine. They wouldn't have to give the million back. And Randi would be delighted to have brought such an amazing thing to light.

 

[paradigm]

As I understand it (and I do not speak for the JREF), that is Randi's judgement call. For instance, if someone claimed to have a telepathic link with another person, and a protocol was agreed, and they passed, they'd win the million. If it was later discovered using fMRI that the person had an extra organ in their brain that gave them this power, and after more study, it was understood how it worked, then fine. They wouldn't have to give the million back. And Randi would be delighted to have brought such an amazing thing to light.

That would make sense. After all if something of that sort was ever found it would no longer be "paranormal" or "supernatural", but would then become another branch of science. Randi's challenge then, in a way (to me), is about opening new branches of science and learning. Assuming, of course, that anyone ever wins. So far we know how that turned out.

 

[fuelair]

A minor part of the problem that Probex has not pointed out in his/her zeal to protect the innocent telepath from the vile money (which he legally can't touch or use) grubbing Randi is that anyone working for the "psychic" would have to be searched thoroughly and be located in a position/place where no signaling could be made to the testee - and if any movement that could be perceived/felt/etc. by the testee occurred, the test would have to be halted. I suspect Randi's people/advisors would be already capable of adding to this list from frauds of the past but I do see that as a problem with Px' suggestion.