Astrology test protocol in progress..

[sadhatter]

Great news! The astrologer is back in business, I thought he was on a retreat in US, but he never got there in the first place. Well, anyways, we're back, and now he has more time on his hands. He agreed to do this:


I will collect 10 birth details.

He will write 10 profiles according to the details.

The 10 volunteers will read all of the profiles and pick the one that they feel is closest to their persona.

His aim is to get 7 correct out of 10.

The volunteers cannot all be sceptics, because of the negative energies, but there can be a few sceptics in the mix.

Minimum age of the volunteers is 25.

Birth certificate is necessary.


I think we are going to lower down the "pass" mark considerably, 7 seems incredible, but that's what he said to me. What would you guys consider to be a meaningful result here? We are currently discussing the best way to choose volunteers. I will keep you all posted.

Well first the believer criteria is suspect, if someone knows this is going to prove their beliefs they may be looking for clues and hints that the psychic left as to what is what ( for example, the psychic could make the capricorn post very capricorny so if a believer sees it they could pick out the capricorn bit thereby making a hit.).

But it does raise some interesting questions , if this does not work with skeptics, wouldn't it be easier just to have 10 skeptics and 10 believers, and then have the psychic tell us who is who based soley on birth dates? If he experiences a psychic block, he should know he is experiencing one, right?

And how much negative energy is okay? It seems your friend is stating that a certain amount is okay, but over a certain amount then his powers don't work. Find out what this amount is beforehand so no excuses can be made after. ( i thought i could handle 3 skeptics, but i could only handle 2 for example. )


Also the minimum age seems suspect. If you give a minimum age, then you limit the possibilities as to who it is going to be, if he has abilities, this should not be effected by different ages.

To be honest it seems your friend is trying to set up a situation in which he has the greatest chance of success. To clarify.

1. Less skeptics mean people can look for things that would give them clues as to what is their reading.

2. Age limitations take out a large segment of the population. He is now only having to give info for 25 plus people. He is doing this to avoid a situation like saying " you will soon marry" and having the applicant be a 2 year old.

3. 7/10 is not that hard to get if you stack the deck like he is trying to do. If he manages to get 2 believers to hone in on the right reading by clues, gets 2 by good wording, and 1 by random chance. He is up to 5 hits already. And if he gets this there will be much chest thumping about " beating chance"

 

[Hokulele]

What type of information is required on the birth certificate?

 

[Kuko 4000]

Well first the believer criteria is suspect, if someone knows this is going to prove their beliefs they may be looking for clues and hints that the psychic left as to what is what ( for example, the psychic could make the capricorn post very capricorny so if a believer sees it they could pick out the capricorn bit thereby making a hit.).

I will check the profiles before I send them forward, this is to eliminate any clues like that and to make sure that there are straightforward claims. In the end, it's pretty much impossible to control the honesty of the volunteer's answers.

But it does raise some interesting questions , if this does not work with skeptics, wouldn't it be easier just to have 10 skeptics and 10 believers, and then have the psychic tell us who is who based soley on birth dates? If he experiences a psychic block, he should know he is experiencing one, right?

And how much negative energy is okay? It seems your friend is stating that a certain amount is okay, but over a certain amount then his powers don't work. Find out what this amount is beforehand so no excuses can be made after. ( i thought i could handle 3 skeptics, but i could only handle 2 for example. )

Yes it does, I just don't want to push stuff like that too much now and offered to choose people who are not sceptics. He replied that as long as they are not all sceptics then it's fine. I wanted to make sure that it was alright and asked again, just for the reasons you raised, and he had no problems with there being a few sceptics in the mix. I may not include any sceptics anyway, remains to be seen, but right now it would seem like the best idea.

Also the minimum age seems suspect. If you give a minimum age, then you limit the possibilities as to who it is going to be, if he has abilities, this should not be effected by different ages.

To be honest it seems your friend is trying to set up a situation in which he has the greatest chance of success. To clarify.

1. Less skeptics mean people can look for things that would give them clues as to what is their reading.

2. Age limitations take out a large segment of the population. He is now only having to give info for 25 plus people. He is doing this to avoid a situation like saying " you will soon marry" and having the applicant be a 2 year old.

2.) I disagree, the age limit is actually very important. I want to give him a good chance to show his skills and to keep everything closer to the real situation. I think I'm gonna raise the minimum age limit to 40 or 50 years. People usually know themselves better when they are older, and having the limit set as low as 25 (or having no limits at all) would considerably limit the scope of the things he can include in his profiles.

3. 7/10 is not that hard to get if you stack the deck like he is trying to do. If he manages to get 2 believers to hone in on the right reading by clues, gets 2 by good wording, and 1 by random chance. He is up to 5 hits already. And if he gets this there will be much chest thumping about " beating chance"

I would think something like 4 or 5 hits is a great result in a test like this? I would again appreciate help from people with more experience in statistics than I have.

 

[Kuko 4000]

What type of information is required on the birth certificate?

The same as before: name, time of birth ("accurate", as discussed earlier) and place of birth.

 

[JoeTheJuggler]

This protocol is OK, but it still has the problems noted earlier. For one, there us the problem of information leakage. The other is that hits or misses are still just judgments. It merely plays the Forer Effect game.

In the Forer Effect, you get about an 80% hit rate even without any real correlation between the "profile" or reading and the person. So if you take that into account, we would expect 8 of 10 right due to the Forer Effect (the tendency of people to see themselves in a profile that is in fact very generic). So if you go with this protocol, it really has to be 10 of 10 for success.

Even so, it only takes the Forer Effect and a little luck.

 

[Kuko 4000]

This protocol is OK, but it still has the problems noted earlier. For one, there us the problem of information leakage. The other is that hits or misses are still just judgments. It merely plays the Forer Effect game.

In the Forer Effect, you get about an 80% hit rate even without any real correlation between the "profile" or reading and the person. So if you take that into account, we would expect 8 of 10 right due to the Forer Effect (the tendency of people to see themselves in a profile that is in fact very generic). So if you go with this protocol, it really has to be 10 of 10 for success.

Even so, it only takes the Forer Effect and a little luck.

I don't understand how the Forer Effect would affect this test. They all have to choose 1 from the available 10 profiles. The Forer Effect would kick in if they would all read and score only one particular profile. If they all end up choosing the most vague or general reading out of the 10 available profiles, that would be only 1 hit for the astrologer.

 

[JoeTheJuggler]

I suggest we go back to the addiction protocol and look for subjects here in the JREF forum. (We were trying to avoid that since the astrologer can read the forum and people who are forum members have their birthdates on their JREF profiles.)

I think with that protocol, there is little chance of that sort of information being of any significance.

 

[JoeTheJuggler]

I don't understand how the Forer Effect would affect this test. They all have to choose 1 from the available 10 profiles. The Forer Effect would kick in if they would all read and score only one particular profile. If they all end up choosing the most vague or general reading out of the 10 available profiles, that would be only 1 hit for the astrologer.

But you're not giving subjects the option of saying none of the profiles are unique to them? Basically they have to assume that one of them uniquely describes them, right?

But yes, I see your point. With the matching thing they're not just relying on the Forer Effect.

ETA: The other problem, from the astrologer's point of view, is that success depends on how well the person knows himself/herself. Even if his "profiles" were perfectly accurate, this protocol would be a measure of how well the subjects can recognize themselves in a profile. That's what I like about the other protocol: no chance of information leakage and the judgment is done completely by the astrologer and no one else.

 

[JoeTheJuggler]

The matching protocol isn't bad, band it's also similar to protocols that have been used for the MDC. I do think the one I've been suggesting is better. The only problem is finding enough subjects that have a history of substance abuse (under a very restrictive definition) who also have a birth certificate with time of birth on it.

If there were some other objective fact easier to come by that the astrologer would agree to (marital status, birth order/number of siblings, occupation or even a combination of all 3 of these), it wouldn't be that difficult to do.

Think he'd agree to matching birth info with current marital status (which has a bit of information leakage related to age--but we could maybe limit subjects to a similar age range) and number of birth order/number of siblings and current occupation?

Then there would be no need for subjects to do the picking and no need to risk information leakage via a the profiles. Just give him the birth info and these 3 item mini profiles, and see if he can match them up. (I have also suggested birth info and the results of some well-known personality profile test too, since most astrologers claim that this birth stuff correlates to personality type.)

 

[sol invictus]

If I'm understanding the protocol, the Forer effect is not relevant here. It makes people likely to say that a random profile they are told is theirs is accurate, but I don't see how it can affect anything when they are asked to choose between two or more such profiles (as in this case).

Each volunteer will pick one out of the ten profiles. If they are choosing randomly, the odds that 7/10 or more will pick their own profile are only about 1/100,000. So if the null hypothesis is that they are choosing randomly, 7 or more hits would rule it out with very high confidence. Even 4 or more hits would give 99% confidence.

The biggest problem I see is what someone pointed out above: the subjects may already know more or less what to expect given their astrological sign and past experience with astrologers, and so might be able to choose the one that most closely matches their expectations.

It's going to be pretty hard to avoid the possibility of such a bias. Ideally you'd pick subjects with no past experience with astrology. Or perhaps you could choose them from a culture, e.g. perhaps China, where astrology is done quite differently?

 

[Kuko 4000]

But you're not giving subjects the option of saying none of the profiles are unique to them? Basically they have to assume that one of them uniquely describes them, right?

I don't know that yet. We might include the options:

A) "I chose this profile, because it is the closest out of all 10, but I still do not recognize myself from it."

and

B) "I chose this profile because I thought it described me well."

If we end up doing this (or having more options), we would have to agree that in the case that someone ticks the option A) but also accidentally chooses the correct profile, it would NOT be considered a hit.

If we don't include something like this there is the danger of false positives (then again, in this case, this would be the equivalent of a chance result). On the other hand, if we do include something like this, he could claim (or at least rationalize it to himself) that the volunteers were too sceptical or dishonest, or something like that. It's a double-edged sword, but I'm actually leaning to having no options at all and let the chance play its role.

Maybe we could include the options in the final analysis, after we (me and the astrologer) have seen the results. If he got any hits, we could call the volunteers who chose the correct profiles and have a discussion about how well they felt the profiles described them and adjust the hits properly.

ETA: The other problem, from the astrologer's point of view, is that success depends on how well the person knows himself/herself. Even if his "profiles" were perfectly accurate, this protocol would be a measure of how well the subjects can recognize themselves in a profile. That's what I like about the other protocol: no chance of information leakage and the judgment is done completely by the astrologer and no one else.

Well, I'm gonna have to think about this, currently I like the profile matching protocol more, since it's closer to the real life situation (which I consider a major point), and I don't see how it would be skewed to his benefit in any meaningful way.

Also, if they can't recognize themselves from a good reading, then his Astrology skill clearly has no role or use in real life situations. Remember, the astrologer could always claim that the volunteers just don't know themselves well enough, even when it comes to substance abuse, "if only they had been more connected with their 'true inner self and potential' this would never had happened", etc. Nevertheless, I agree, that the substance protocol leaves less wiggle room in this respect.

Anyways, this is how I see it at the moment, always open for improving my understanding.

 

[Kuko 4000]

The biggest problem I see is what someone pointed out above: the subjects may already know more or less what to expect given their astrological sign and past experience with astrologers, and so might be able to choose the one that most closely matches their expectations.

It's going to be pretty hard to avoid the possibility of such a bias. Ideally you'd pick subjects with no past experience with astrology. Or perhaps you could choose them from a culture, e.g. perhaps China, where astrology is done quite differently?

I agree, the best I can think to eliminate this is to choose participants with no prior knowledge of astrology. Even then, there is no way for me to know that for sure, but that is a pretty minor danger in my opinion. In addition to this, I will check the profiles myself before I will hand them to the participants. My aim with this is to edit out any obvious clues that could help the participants connect with the correct profile.

 

[AvalonXQ]

Why not just get all the volunteers from a single sun sign?

 

[sadhatter]

I will check the profiles before I send them forward, this is to eliminate any clues like that and to make sure that there are straightforward claims. In the end, it's pretty much impossible to control the honesty of the volunteer's answers.





Yes it does, I just don't want to push stuff like that too much now and offered to choose people who are not sceptics. He replied that as long as they are not all sceptics then it's fine. I wanted to make sure that it was alright and asked again, just for the reasons you raised, and he had no problems with there being a few sceptics in the mix. I may not include any sceptics anyway, remains to be seen, but right now it would seem like the best idea.





2.) I disagree, the age limit is actually very important. I want to give him a good chance to show his skills and to keep everything closer to the real situation. I think I'm gonna raise the minimum age limit to 40 or 50 years. People usually know themselves better when they are older, and having the limit set as low as 25 (or having no limits at all) would considerably limit the scope of the things he can include in his profiles.




I would think something like 4 or 5 hits is a great result in a test like this? I would again appreciate help from people with more experience in statistics than I have.

So first off, if your afraid to " push" stuff, then your not in the correct mindset. The problem is your leaving many holes for the psychic use. For example if he does not say how many skeptics can be in the mix, then he has an out if he fails. These things need to be quantified before the test, so that no excuses can come about. And if your worried about politeness then your not going to get that done. And the comment about " not all skeptics" so 9/10 is okay? It is already starting to not add up.

The problem is, and i mean no offense by this, at all, your simply not seeing what he is doing in order to stack the deck. There is no real way to eliminate the clues he would give, as someone who used to be really into this kind of thing, i could pick out an astrological reading for my sign fairly easily.

Again, if he is claiming skeptics kill his powers, put this to the test first. Give him a list of skeptics and believers birthdates and have him tell you which is which. If he is being blocked by the skeptics it should be very easy to tell who is who.

"2.) I disagree, the age limit is actually very important. I want to give him a good chance to show his skills and to keep everything closer to the real situation. I think I'm gonna raise the minimum age limit to 40 or 50 years. People usually know themselves better when they are older, and having the limit set as low as 25 (or having no limits at all) would considerably limit the scope of the things he can include in his profiles."

bolding mine.

I have do deal with this comment separately, and again no offense intended, i am simply calling them as i see them.

It seems at this point you are attempting to help him.

Of course it is going to limit what he can put in his readings, because he has a lot less sure bets. The higher limit you put on the age, the easier it is to do a reading. I have studied cold reading myself, and this is one of the easiest ways to make sure your going to come off well.

This is my biggest problem with the protocol, you are eliminating him being able to be wrong in the following ways.

1) He knows that the participants will not generally have "new" taste in media or music. In effect he has a demographic of his audience.

2) He has a huge chance of sucess with any dead relations he mentions.

3) He knows the person is not A) dead or B) an infant. So he knows no surprises are going to come his way in this regard.

4) He knows what major world events would have effected the people in their childhood.

Honestly i could make this list go on and on, but the worst mistake you can make is letting him slide in an age limitation.

And from a skeptical standpoint, you should throw in a dead person or two, if he gives a reading for a dead man, you can be pretty much sure he is full of it. And if he has powers, he should be able to sense said person is dead.

If you ignore every other piece of advice i give, i must stress the age limit is a huge red flag to me that there is some Woo going on. I suggest reading some books on cold reading so you can understand the type of things he will be doing in order to give himself greater odds.

And finally 5/10 with this sample size is not a good result, as i showed it has a great chance of coming up by chance if you allow him to stack the deck as you have been. And more to the point , if he is saying he can get 7/10 then 6/10 isn't good, 5/10 isn't good, only 7/10 and above is good. If the man cannot do what he claims, he cannot do what he claims. And more to the point, if he is real, why should he be getting any wrong whatsoever?

You seem to much a friend to this person versus an unbiased skeptic, my advice is not to let him run the show, i understand it can be hard to be so hardline with a friend, but if your not all your doing is another faulty test. And on the flip side as a previous woo artist myself, i can see no less than 5 excuses i could make for failure that would fly with the woo crowd.

 

[sadhatter]

I agree, the best I can think to eliminate this is to choose participants with no prior knowledge of astrology. Even then, there is no way for me to know that for sure, but that is a pretty minor danger in my opinion. In addition to this, I will check the profiles myself before I will hand them to the participants. My aim with this is to edit out any obvious clues that could help the participants connect with the correct profile.

If you edit his information, the the excuse of " well of course they couldn't tell who was who, you screwed it up" can and will be used.

 

[Kuko 4000]

So first off, if your afraid to " push" stuff, then your not in the correct mindset. The problem is your leaving many holes for the psychic use. For example if he does not say how many skeptics can be in the mix, then he has an out if he fails. These things need to be quantified before the test, so that no excuses can come about. And if your worried about politeness then your not going to get that done. And the comment about " not all skeptics" so 9/10 is okay? It is already starting to not add up.

The thing is, I want to give him a fair chance to show his skills, as discussed (way) earlier, if he says he can't do his magic when there are sceptics in the mix, then so be it, no problems at all. It would still be magic. And I'm absolutely fine with that. We both need to be 100% happy about the protocol and I am 100% happy with excluding sceptics from the mix if that makes his magic work.

And, no, 9 / 10 would not be ok, we already agreed that if there is going to be sceptics in the mix it would only be 1-3. But like I said, I will probably keep the test sceptic-free.

The problem is, and i mean no offense by this, at all, your simply not seeing what he is doing in order to stack the deck. There is no real way to eliminate the clues he would give, as someone who used to be really into this kind of thing, i could pick out an astrological reading for my sign fairly easily.

Really? What about only including people who have no knowledge of astrology in the first place as the participants? I find that to be reasonably safe for a test like this.

Again, if he is claiming skeptics kill his powers, put this to the test first. Give him a list of skeptics and believers birthdates and have him tell you which is which. If he is being blocked by the skeptics it should be very easy to tell who is who.

I don't see the need for this, I can easily eliminate the evil sceptics from the participant list and we are both happy.

bolding mine.

I have do deal with this comment separately, and again no offense intended, i am simply calling them as i see them.

It seems at this point you are attempting to help him.

Of course it is going to limit what he can put in his readings, because he has a lot less sure bets. The higher limit you put on the age, the easier it is to do a reading. I have studied cold reading myself, and this is one of the easiest ways to make sure your going to come off well.

This is my biggest problem with the protocol, you are eliminating him being able to be wrong in the following ways.

1) He knows that the participants will not generally have "new" taste in media or music. In effect he has a demographic of his audience.

2) He has a huge chance of sucess with any dead relations he mentions.

3) He knows the person is not A) dead or B) an infant. So he knows no surprises are going to come his way in this regard.

4) He knows what major world events would have effected the people in their childhood.

Honestly i could make this list go on and on, but the worst mistake you can make is letting him slide in an age limitation.

I think you're mistaken here, as far as I can tell, this test is far removed from a 1 to 1 situation where you could use Cold Reading skills and from the situation where the Forer Effect would kick in.

1) This is the same for all the participants.

2) This is the same for all the participants.

3) This is the same for all the participants.

4) I will do my best to edit all the age-related hints from the reading.

All of the participants are given 10 profiles and they are all going to have to choose one profile that describes them the best. I just don't see how your objections would matter in this test.

And from a skeptical standpoint, you should throw in a dead person or two, if he gives a reading for a dead man, you can be pretty much sure he is full of it. And if he has powers, he should be able to sense said person is dead.

This would be too far removed from the real situation, which is a major point for me. And I'm already pretty sure he has no magic powers, as sure as I can be of anything else really.

If you ignore every other piece of advice i give, i must stress the age limit is a huge red flag to me that there is some Woo going on. I suggest reading some books on cold reading so you can understand the type of things he will be doing in order to give himself greater odds.

Due to the protocol of this test, I don't see how cold reading could play a meaningful part. I would be grateful if you could show me the problems in my answers regarding Cold Reading (the 4 points above).

And finally 5/10 with this sample size is not a good result, as i showed it has a great chance of coming up by chance if you allow him to stack the deck as you have been. And more to the point , if he is saying he can get 7/10 then 6/10 isn't good, 5/10 isn't good, only 7/10 and above is good. If the man cannot do what he claims, he cannot do what he claims. And more to the point, if he is real, why should he be getting any wrong whatsoever?

You seem to much a friend to this person versus an unbiased skeptic, my advice is not to let him run the show, i understand it can be hard to be so hardline with a friend, but if your not all your doing is another faulty test. And on the flip side as a previous woo artist myself, i can see no less than 5 excuses i could make for failure that would fly with the woo crowd.

Well, he says that even the best astrologer can achieve "only" a 75% accuracy in their readings, that's why he would get some of them wrong even if he would perform at his best level. I am just not convinced yet that your objections are valid in this test. Don't get me wrong, I will think about them seriously and more, thanks for your input.

If you edit his information, the the excuse of " well of course they couldn't tell who was who, you screwed it up" can and will be used.

As discussed (way) earlier, he will not have this "out" since after I have edited his profiles, I will send them back to him for approval. We will send the profiles to the participants only after we are both 100% happy about them.

 

[Kuko 4000]

Why not just get all the volunteers from a single sun sign?

I'm wondering what good would this do? The readings would probably end up being very similar to each other, right? And, as far as I can tell, this would only complicate matters unnecessarily. Maybe I'm missing your point.

 

[Kuko 4000]

Each volunteer will pick one out of the ten profiles. If they are choosing randomly, the odds that 7/10 or more will pick their own profile are only about 1/100,000. So if the null hypothesis is that they are choosing randomly, 7 or more hits would rule it out with very high confidence. Even 4 or more hits would give 99% confidence.

For the small chance that I might learn something math-related for longer than a minute or two, would you mind opening up the calculation, I'm not quite sure how you ended up with those results? I thought getting 7 / 10 would be even more improbable. Thanks!

 

[sol invictus]

Why not just get all the volunteers from a single sun sign?

That would reduce but not eliminate the problem, since they might have had a more detailed reading done in the past.

I don't know that yet. We might include the options:

A) "I chose this profile, because it is the closest out of all 10, but I still do not recognize myself from it."

and

B) "I chose this profile because I thought it described me well."

If we end up doing this (or having more options), we would have to agree that in the case that someone ticks the option A) but also accidentally chooses the correct profile, it would NOT be considered a hit.

If we don't include something like this there is the danger of false positives (then again, in this case, this would be the equivalent of a chance result). On the other hand, if we do include something like this, he could claim (or at least rationalize it to himself) that the volunteers were too sceptical or dishonest, or something like that. It's a double-edged sword, but I'm actually leaning to having no options at all and let the chance play its role.

I don't see why two options are necessary. I think they'll just add complication with no benefit and muddle the results.

Just have them each pick the single best match and set the threshold for success based on a null hypothesis that they're choosing randomly. Then do your best to eliminate anything that might give them hints. If you go with 7/10, it's 1/100,000 by chance alone - that's not going to happen.

 

[sol invictus]

For the small chance that I might learn something math-related for longer than a minute or two, would you mind opening up the calculation, I'm not quite sure how you ended up with those results? I thought getting 7 / 10 would be even more improbable. Thanks!

The easiest way is to use an online calculator. Enter .1, 10, and 7 in the first three boxes and click "calculate". The odds you want are the cumulative probability x>=7, since you'd consider 7 or more hits a success.

If you want to know how to get the result I can walk you through it. Roughly it's .1^7 (that's the odds that a given group of 7 all picked correctly) times the number of ways you can choose a group of 7 out of 10, which is 120. That gives you about 1/100,000. The odds of getting 8/10 or more are so small they hardly matter.