Merged Bem's latest experiments / Researcher says we can see the future

[Elaedith]

As them what the hell they think is going on. Tell them you're so fascinated to learn that you're willing to try some replication.

~~ Paul

But then if a skeptic fails to replicate it must be because skepticism inteferes with the effect.

Really though, its hardly necessary for the experimenter to be involved in this type of study. You just prepare the computer scripts, put the instructions on the computer, shut somebody alone in a room and wait. Or it can even be done online with the experimenter not around at all.

 

[Jeff Corey]

Even smallish sample sizes can be sufficient to establish an effect with an appropriate research design. The much bigger problem is that the experimental design is not set up to test whether or not there is an effect from psi, but rather whether or not heterogeneous data can be grouped and regrouped in ways that lead to between-group differences. Then if any between-group differences are demonstrated, a failure to be sufficiently clever to account for those differences post hoc is meant to represent 'psi'.

Linda

Yes, in my world, real effects don't need a huge number of participants (PC for subjects) and a baroque design.
I am very busy right now - midterms, papers to grade and the Rally for Sanity to plan for.
Linda was kind enough to PM me a heads up about this publication a week ago. It wasn't clearly written and I think that there are ways to do it in such a way to yield the results more explicable.
I do like the online experiment idea, I use the APA psychology lab online. http://opl.apa.org/

 

[fls]

But the entire point of random is that it does not coincide. If the random generator is truly random, there is no need to worry about that. Yes maybe for one participant 60% coincides and for the other 40% but they average each other out because it is random.

Right. But this is the third week in a row I've won money on the lottery.

There is a difference between what might happen and what has happened.

About the sample sizes, with cases over n=30 (per cell), normal distributions is an assumption that is not strict (the tests are robust against violation of this assumtion).

The analyses for which the tests are robust are not those we are concerned about. We are concerned with how each baseline is generated for each participant, not with the data collected from all the participants. The number of observations are those which go into each individual data point. In the case of the first experiment, it will be "12" or "18". In the case of the last experiment, it may be "1".

A small thing I noted b.t.w. is that in experiment 1 it is claimed that positive images are not enough to lead to "psi" but later experiments (the backwards priming ones) show an effect of positive primes

Yes. It isn't unusual for parapsychologists to fail to replicate their own findings. This doesn't seem to concern them, as the findings will be taken to demonstrate some different aspect of psi instead.

Linda

 

[roger]

Yes. It isn't unusual for parapsychologists to fail to replicate their own findings. This doesn't seem to concern them, as the findings will be taken to demonstrate some different aspect of psi instead.

Linda

Yes, I remember another paper Paul posted here a few years ago. I don't remember if it was Bem's or not. But the test did not demonstrate any statistical effect, so he/they decided that there must be a negative psi effect, divided the data into groups that did better than average, groups that did worse than average, and showed 'statistical significance' on those subgroups!

That's basically all I'm seeing in this paper. Something doesn't show an effect, randomly change something, and claim success when this trial is positive. No going back to actually test whether factor X is actually an influence, no second experiment run to see if you can replicate the results, just off to the next inconclusive experiment with entirely different setup while claiming factor X does whatever you assumed it does. Oy vey.

 

[squig]

The analyses for which the tests are robust are not those we are concerned about. We are concerned with how each baseline is generated for each participant, not with the data collected from all the participants. The number of observations are those which go into each individual data point. In the case of the first experiment, it will be "12" or "18". In the case of the last experiment, it may be "1".

Linda

Most of the time the baserates are chance, definately for experiment 8 & 9. (though I think there are other issues in those experiments)

Right. But this is the third week in a row I've won money on the lottery.

There is a difference between what might happen and what has happened.

But the claim is that the statistics / method was wrong.

I am not trying to be an ass, but a lot of people here claim that the method is wrong, or that he mixed and matched groups leading to sig. results.
I am not blind for some of the dodgy tricks he pulls, i.e., experiment 1 using 5 groups finding sig results for one and (probably) post-hoc rationalising it.

However, I fail to see the problems with most of the studies. Of course I agree that his theory is nonsense. At the same time though, I feel that people are dismissing his experiments here on the basis of ill-formed arguments.

Assuming that the psi-argument is nonsense, & assuming that the author is not intentionally lying; what can be said about the methods, which more reasonable explanation can account for the findings?


Since it is easy to complain and not contribute anything Ill just give my thoughts on the experiments themselves quickly:

experiment 1: no reasoning for the different effects of the different pictures. If rationalised post-hoc, the sig. goes up to .32(ish)

experiment 2: Even though Bem uses odd adjustments, without the adjustments he still finds sig. results. In all my ignorance I can not imagine any fault in the method that can lead to this.
Theorie-wise (if you can even talk about that) why do positive pictures all of a sudden work here?

experiment 3: Expectations / lay theories could lead to deviations from chance i think (though Im not 100% sure), that's all I have really I can not come up with more.
theory-wise: again why are both positive and negative grouped if the claim later in the paper is that people react to positive stuff?

experiment 4: see 3
(also the means are remarkably higher)

experiment 5: perhaps the random-generator is off?
theory-wise; why should positive work and neutral not?
(also noticed; everything he hopes is n.s. is all of a sudden tested two-tailed)

experiment 6: seems to me some sort of post-hoc rationalisation of the null-result or is there a reason Im missing why erotic should not work?

experiment 7: why a correlation with lower and not higher?

experiment 8: I really do not understand why the scoring system is so complicated, this makes me somewhat suspicious. I do not understand why raw scores could not have been used. However, the found result seems to be pretty large and the only way the scoring system could account for that is if Bem fished for results (breaking the assumption of him being an honost man ). More than that I can not find.

experiment 9: see 8

Yes, I remember another paper Paul posted here a few years ago. I don't remember if it was Bem's or not. But the test did not demonstrate any statistical effect, so he/they decided that there must be a negative psi effect, divided the data into groups that did better than average, groups that did worse than average, and showed 'statistical significance' on those subgroups!

That's basically all I'm seeing in this paper. Something doesn't show an effect, randomly change something, and claim success when this trial is positive. No going back to actually test whether factor X is actually an influence, no second experiment run to see if you can replicate the results, just off to the next inconclusive experiment with entirely different setup while claiming factor X does whatever you assumed it does. Oy vey.

Like above; can you point out where this happens because I just do not see it (cept for exp 1&6&7)

I personally do not believe Bems rationale, nor do I believe he has been outright lying. Furthermore, I do not like that he has fiddled with the data and the representation of his data but at the same time that is not enough to explain the findings. Therefore, there must be something wrong with the method. I just can not find it....

 

[Paul C. Anagnostopoulos]

I agree with squig: It's too easy to blow off the results due to a misunderstanding of the methodology. We have to analyze an experiment carefully to see if there are methodological flaws, or perhaps missing information about the protocol, or a mismatch between the statistical model and the actual protocol, etc. If we can't find anything then we can call fraud or delusion, or, more likely, we just have to sit back and wait.

The protocols appear to be simple and easy to replicate. We could do it online. It's being published in a well-known journal. I think we will see replication attempts. I'll certainly watch out for them and post them here.

No word from Bem yet.

~~ Paul

 

[fls]

Most of the time the baserates are chance, definately for experiment 8 & 9. (though I think there are other issues in those experiments)

Based on the distribution from which the sample was drawn, not based on the distribution drawn. And we know that these are different, since the few times they are analyzed in comparison, there are slight differences.

But the claim is that the statistics / method was wrong.

Yes, a claim which Bem recognizes as worthy of consideration.

I am not trying to be an ass, but a lot of people here claim that the method is wrong, or that he mixed and matched groups leading to sig. results.
I am not blind for some of the dodgy tricks he pulls, i.e., experiment 1 using 5 groups finding sig results for one and (probably) post-hoc rationalising it.

However, I fail to see the problems with most of the studies. Of course I agree that his theory is nonsense. At the same time though, I feel that people are dismissing his experiments here on the basis of ill-formed arguments.

So you are simply ignoring the concern Ersby and I raised, which was the general lack of control groups?

Assuming that the psi-argument is nonsense, & assuming that the author is not intentionally lying; what can be said about the methods, which more reasonable explanation can account for the findings?

Well, these experiments do have the characteristics which increase the probability of false-positive results - independent testing, small studies, small effects, multiple tested relationships, flexibility in design, definitions, outcomes and analytical modes. Although, I suppose that this suggestion is not reasonable since it didn't come from you.

Linda

 

[fls]

I agree with squig: It's too easy to blow off the results due to a misunderstanding of the methodology. We have to analyze an experiment carefully to see if there are methodological flaws, or perhaps missing information about the protocol, or a mismatch between the statistical model and the actual protocol, etc. If we can't find anything then we can call fraud or delusion, or, more likely, we just have to sit back and wait.

~~ Paul

I'm sorry, but are you seriously suggesting that it is more likely that Bem committed fraud than that the methodological issues already identified could contribute to the results?

Linda

 

[squig]

I agree with squig: It's too easy to blow off the results due to a misunderstanding of the methodology. We have to analyze an experiment carefully to see if there are methodological flaws, or perhaps missing information about the protocol, or a mismatch between the statistical model and the actual protocol, etc. If we can't find anything then we can call fraud or delusion, or, more likely, we just have to sit back and wait.

The protocols appear to be simple and easy to replicate. We could do it online. It's being published in a well-known journal. I think we will see replication attempts. I'll certainly watch out for them and post them here.

No word from Bem yet.

~~ Paul

Problem is that you need a lot more experiments to publish failures to replicate than you need to publish results :/

Based on the distribution from which the sample was drawn, not based on the distribution drawn. And we know that these are different, since the few times they are analyzed in comparison, there are slight differences.



Yes, a claim which Bem recognizes as worthy of consideration.



So you are simply ignoring the concern Ersby and I raised, which was the general lack of control groups?



Well, these experiments do have the characteristics which increase the probability of false-positive results - independent testing, small studies, small effects, multiple tested relationships, flexibility in design, definitions, outcomes and analytical modes. Although, I suppose that this suggestion is not reasonable since it didn't come from you.

Linda

Im not sure why you are making such snide comments, I do not think it is necessary t.b.h.
I am not ignoring what you guys have stated, I even asked for clarification. I just still do not understand the claim. You talk about the lack of control-groups, I see multiple control groups. In the experiments where he does not use a control he tests against chance. You do not need a control group to test whether the data were truly random, you just need the data.

If you had read my posts carefully you would have seen that I did mention all of the stuff you mention above, however you would have also read my claim that this alone is not enough to explain the results.

So we can all either sit in our ivory towers claiming that Bem was wrong just because, or actually try and disentangle his experiments and truly find out what lead to which results.

Once more, I am not ignoring your comments or claiming them to be wrong, I just do not see how they can explain everything. If you think otherwise, then please do something similar as me and adress the methodolical flaw that has lead to the results per experiment and enlighten me.

 

[Paul C. Anagnostopoulos]

I'm sorry, but are you seriously suggesting that it is more likely that Bem committed fraud than that the methodological issues already identified could contribute to the results?

No, but I do think we have the tendency to identify what might be flaws and then simply assume they are flaws.

~~ Paul

 

[fls]

Im not sure why you are making such snide comments, I do not think it is necessary t.b.h.

I didn't mean them to be snide. I just got a kick out you chiding me for having ill-formed arguments based on, as far as I could tell, the problem that you thought a different set of ill-formed arguments applied. Hence the smilie.

I am not ignoring what you guys have stated, I even asked for clarification. I just still do not understand the claim. You talk about the lack of control-groups, I see multiple control groups. In the experiments where he does not use a control he tests against chance. You do not need a control group to test whether the data were truly random, you just need the data.

Well, you need a control group to establish the baseline in the absence of whatever effect you want to test. The closest he comes to this is in experiments 8 and 9. Otherwise, all he has managed to establish is that some of these groupings are unlikely if they are obtained by chance. Unfortunately, this doesn't tell us if chance is still the most likely explanation (since known alternate explanations are still present (like testing multiple relationships)) and it tells us nothing about what we'd see if there was something present we might call 'psi'. We are just supposed to assume that the likelihood these results would appear were it chance is somehow complementary to the likelihood that these results would appear were it 'psi', which is fallacious, of course.

We're not interested in what sort of results we can attribute to chance, but what sort of results we can attribute to psi. Continuing to test the former doesn't really help us with the latter.

If you had read my posts carefully you would have seen that I did mention all of the stuff you mention above, however you would have also read my claim that this alone is not enough to explain the results.

How do you know, considering that we were not given enough information to make that call, and that the tests weren't designed so as to allow us to rule-out those effects?

So we can all either sit in our ivory towers claiming that Bem was wrong just because, or actually try and disentangle his experiments and truly find out what lead to which results.

Once more, I am not ignoring your comments or claiming them to be wrong, I just do not see how they can explain everything. If you think otherwise, then please do something similar as me and adress the methodolical flaw that has lead to the results per experiment and enlighten me.

Why is it necessary to explain everything? Let's pretend that we can and have managed to deal with every potential explanation you and Bem are clever enough to come up with. What conclusions can you draw about whatever it is that explains the results from an inability to explain the results?

Linda

 

[fls]

No, but I do think we have the tendency to identify what might be flaws and then simply assume they are flaws.

~~ Paul

How would they not be flaws? If we know that outcomes differ in the presence of blinding, then a lack of blinding is a flaw regardless of whether you can find this effect post hoc. If we don't know whether blinding has an effect a priori, then a lack of blinding is still treated as a potential flaw until you can demonstrate otherwise - not by looking for the effects from a lack of blinding post hoc, since this can be insensitive, but by performing tests with and without blinding and comparing the results.

I agree that we don't know whether or not the results will be different if performed in the absence of flaws which can produce those results. And we also don't know if the appearance of some of these flaws is because of the way they are described. For example, Bem doesn't specifically state that the analyses were performed blind. This may be because they weren't or because they were but describing it had a low priority.

Linda

 

[Paul C. Anagnostopoulos]

How would they not be flaws? If we know that outcomes differ in the presence of blinding, then a lack of blinding is a flaw regardless of whether you can find this effect post hoc. If we don't know whether blinding has an effect a priori, then a lack of blinding is still treated as a potential flaw until you can demonstrate otherwise - not by looking for the effects from a lack of blinding post hoc, since this can be insensitive, but by performing tests with and without blinding and comparing the results.

I agree that we don't know whether or not the results will be different if performed in the absence of flaws which can produce those results. And we also don't know if the appearance of some of these flaws is because of the way they are described. For example, Bem doesn't specifically state that the analyses were performed blind. This may be because they weren't or because they were but describing it had a low priority.

We have nothing but agreement here. I simply think we need to be careful to be a bit skeptical about our own somewhat quick analysis of these papers and our assumptions about flaws.

That said, I'm really uncomfortable about how well these experiments turned out. It just seems almost too good to be true. I'm really interested in future replication attempts.

~~ Paul

 

[fls]

We have nothing but agreement here. I simply think we need to be careful to be a bit skeptical about our own somewhat quick analysis of these papers and our assumptions about flaws.

I agree. I'm not saying the results can be dismissed. I'm just saying that we cannot make much out of them yet.

That said, I'm really uncomfortable about how well these experiments turned out. It just seems almost too good to be true. I'm really interested in future replication attempts.

~~ Paul

Ah yes. The perfect set-up to occupy the field for another 20 years before it gives up and moves on to the next big thing. The next time a grad student, or even an undergrad student, comes to Jeff Corey or Elaedith looking for a new project, they'll suggest giving this a try. Multiply that by a hundred and add in the data massaging described in the paper plus variations in design, outcomes, and analysis, and you can probably guarantee at least 20% will provide you with 'intriguing' results. Those will get written up and published. Some of the unremarkable results will get published as well, but there's a fairly limited market for that, and it's easy enough to drop an undergrad project once it has achieved its desired effect (the attention of a prof). Interest will wane amongst psychologists as they notice that most of the time the experiment doesn't demonstrate any unusual results and they feel a bit guilty about the massage. But parapsychologists still on the bandwagon will start performing meta-analyses, 'proving' that an effect is present and using their invalid fail-safe N to grossly underestimate the effects of the dropped and unpublished studies. They will continue to cling to the idea, chastising the close-minded psychologists for failing to see what is so plain to them, but even parapsychologists will eventually notice that the 'decline effect' has come into play. By that time someone will have come up with a whole new set of experiments, chosen for their ability to generate some 'positive' results, testing some other aspect of psi, that will become the New Big Thing, leaving parapsychologists to chortle at the naivety of the grad student asking, "if they worked so well, how come we no longer use Rhine cards/remote viewing/ganzfeld/RNG's?"

But this time it'll be different.

Linda

 

[Garrette]

I agree. I'm not saying the results can be dismissed. I'm just saying that we cannot make much out of them yet.



Ah yes. The perfect set-up to occupy the field for another 20 years before it gives up and moves on to the next big thing. The next time a grad student, or even an undergrad student, comes to Jeff Corey or Elaedith looking for a new project, they'll suggest giving this a try. Multiply that by a hundred and add in the data massaging described in the paper plus variations in design, outcomes, and analysis, and you can probably guarantee at least 20% will provide you with 'intriguing' results. Those will get written up and published. Some of the unremarkable results will get published as well, but there's a fairly limited market for that, and it's easy enough to drop an undergrad project once it has achieved its desired effect (the attention of a prof). Interest will wane amongst psychologists as they notice that most of the time the experiment doesn't demonstrate any unusual results and they feel a bit guilty about the massage. But parapsychologists still on the bandwagon will start performing meta-analyses, 'proving' that an effect is present and using their invalid fail-safe N to grossly underestimate the effects of the dropped and unpublished studies. They will continue to cling to the idea, chastising the close-minded psychologists for failing to see what is so plain to them, but even parapsychologists will eventually notice that the 'decline effect' has come into play. By that time someone will have come up with a whole new set of experiments, chosen for their ability to generate some 'positive' results, testing some other aspect of psi, that will become the New Big Thing, leaving parapsychologists to chortle at the naivety of the grad student asking, "if they worked so well, how come we no longer use Rhine cards/remote viewing/ganzfeld/RNG's?"

But this time it'll be different.

Linda

Nom nom.

 

[Paul C. Anagnostopoulos]

I've not heard back from Daryl Bem, so I sent a second request.

~~ Paul

 

[blobru]

I've not heard back from Daryl Bem, so I sent a second request.

~~ Paul

Let's hope the second request acts as a retroactive prime, to increase the likelihood of a response to the first.

 

[Paul C. Anagnostopoulos]

Let's hope the second request acts as a retroactive prime, to increase the likelihood of a response to the first.

Damn, I forgot to attach an erotic photo!

~~ Paul

 

[blobru]

There -- a google of "BEM", "pulp", & "babe" yields these anatomically unlikely Bug-Eyed Monster pulp scifi covergirl erotic trysts: which, according to the professor's latest data, have just increased our chances of a positive response by ~3.5%, I think (so long as Bem is an extraverted female, that is...).

 

[Arouet]

Here's a paper that has apparently been submitted for publication to the JPSP critically looking at the Bem paper. One proponent on Skeptiko has already called it "a load of crap dressed up in a nice pdf package", but I thought that I would post it anyway, to see what the thoughts are here.

http://dl.dropbox.com/u/1018886/Bem6.pdf