Astrology test protocol in progress..

[Kuko 4000]

Perhaps this could pave the way for a more stringent test, these are not very steep odds.

I'm not proceeding until I get a good idea about the maths, anyone willing to crunch the numbers?

As someone pointed out, if the volunteers were knowledgeable about astrology, or simply has had their chart run before, it would void the test, they could recognize the appropriate response for the data they have given.

If the volunteers are willing to lie about the claims, how could I control anyone from lying that they are not knowledgeable about astrology in the first place? If they answer the claims honestly, I don't know how a volunteer could "cheat".

 

[Marcus]

If the volunteers are willing to lie about the claims, how could I control anyone from lying that they are not knowledgeable about astrology in the first place? If they answer the claims honestly, I don't know how a volunteer could "cheat".

I guess you couldn't control for that, it's a weakness in the protocol, but you are not going for a million dollars here, nothing wrong with running this kind of test. You could at least ask them whether they had their chart run before, though, and take their word for it.

There are plenty of people here who would be happy to do the math, if one doesn't show up on this thread, browse one of the recent challenge threads and find one to pm.

 

[Hokulele]

I guess you couldn't control for that, it's a weakness in the protocol, but you are not going for a million dollars here, nothing wrong with running this kind of test. You could at least ask them whether they had their chart run before, though, and take their word for it.

I wouldn't worry too much about this, as I am guessing the astrologer in question is using Vedic charts as opposed to the more common Western charts. I agree, this test won't be conclusive in any sense, but it may demonstrate to the astrologer the difference between giving readings in a face-to-face interview (cold/warm reading, Forer effect, etc.) and one that is a bit more dispassionate. There have been any number of paranormal believers who have been shocked at how the techniques that work so well in a personal setting are far less effective without direct communication.

 

[Kuko 4000]

I wouldn't worry too much about this, as I am guessing the astrologer in question is using Vedic charts as opposed to the more common Western charts. I agree, this test won't be conclusive in any sense, but it may demonstrate to the astrologer the difference between giving readings in a face-to-face interview (cold/warm reading, Forer effect, etc.) and one that is a bit more dispassionate. There have been any number of paranormal believers who have been shocked at how the techniques that work so well in a personal setting are far less effective without direct communication.

I recently learned that he has done over a hundred readings already and most of them by phone or face to face. I didn't want to get started on cold and hot reading yet, but after the test would be a great opportunity for that.

 

[sol invictus]

I think this protocol is flawed in a few ways.

One problem is that knowing the year and country of birth could allow him to make some guesses that resonate with people from that generation and country. Since your group of volunteers might be very international and diverse in age, that could be rather important. It would be better to control for it by using people that are all roughly the same age and from the same region.

Another problem is statistical. You're requiring that the profile the astrologer made for person X has to receive more hits from person X than from anyone else. I think that's probably too stringent. For example, suppose you had 100 people rather than 10 (but you still only had 3 profiles). Then it would be rather likely some other person would record more hits than person X, even if the profile is a pretty accurate description of person X. Generally, it's better to avoid statistics for which a single outlier can have a large effect.

If you do use this protocol, rather than imposing an arbitrary standard for success as you've done, I think you should require that the results exceed some probability cutoff; that is that the test is a success if the odds it worked as well as it did by random chance are less than 5% (for example). To use that you'll need to calculate the odds. To do that you'll need some data from the test (like the average number of hits and its standard deviation) and you'll need to make some assumptions (that hits obey a binomial distribution, for example).

 

[Kuko 4000]

I think this protocol is flawed in a few ways.

One problem is that knowing the year and country of birth could allow him to make some guesses that resonate with people from that generation and country. Since your group of volunteers might be very international and diverse in age, that could be rather important. It would be better to control for it by using people that are all roughly the same age and from the same region.

I'm trying to eliminate this by checking and correcting the profiles before sending them to the volunteers. The astrologer has agreed with this, we will send the profiles back and forth until they have no such clues and we are both 100% happy. About the age thing, I'm pretty sure I will set the age limit up to 40 or 50 so that he can talk about age and year related things without worrying too much. This will be better for both the quality of the test and for the astrologer to make more detailed guesses.

I don't see the problem after I've checked the profiles. The profiles will also become public after the test is finished and everyone can point out if I've made any errors in judgement that would benefit the astrologer.

If you do use this protocol, rather than imposing an arbitrary standard for success as you've done, I think you should require that the results exceed some probability cutoff; that is that the test is a success if the odds it worked as well as it did by random chance are less than 5% (for example). To use that you'll need to calculate the odds. To do that you'll need some data from the test (like the average number of hits and its standard deviation) and you'll need to make some assumptions (that hits obey a binomial distribution, for example).

Do you mean that when the volunteers have chosen the profile that they got most hits in, I, or whoever works as the middle man, would then see how the points compare with the overall point distribution, and if the target persons have significantly higher rate of "hits" than the average the test is a success? Damn the language barrier, it makes certain kind of thinking very difficult, especially when the field is pretty much unknown to me, apologies for that.

 

[JoeTheJuggler]

What he should NOT include in the profiles:

Ages, dates, countries.

Or references to typical zodiac sun signs or cognates. Most people know or recognize those signs and words. It's not an easy task--he can't say "because the Moon is in Libra" or even something like "you have a tendency to look for balance in chaotic situations". You've got to be on the lookout for words like "bull-headed" and so on.

It's an easy source of information leakage.

You could get around this by choosing 3 that all fall within the same sun sign.

ETA: I also dislike protocols that depend on the subjects' evaluation of the profiles. It seems that success or failure is not in the hands of the astrologer.

I would rather something like this: have all the volunteers take a standard personality profile test. Give the anonymous results to the astrologer and the anonymous birth information. The astrologer must associate birth info with the personality profile test results (with numbers for a success that would rule out chance to some level of confidence).

 

[AvalonXQ]

I would rather something like this: have all the volunteers take a standard personality profile test. Give the anonymous results to the astrologer and the anonymous birth information. The astrologer must associate birth info with the personality profile test results (with numbers for a success that would rule out chance to some level of confidence).

But then the results depend on the accuracy of the personality profile, which has its own issues.
This is why the Carlson experiment failed to show anything conclusive -- they used a personality test where, at the end, people couldn't even correctly identify which results were theirs.

 

[Neally]

I'm trying to overcome this by marking the exact claims in the profile, and asking the volunteers to rate the claims as either a hit or a miss.

Perhaps you should get a sample profile that he has completed. If it's filled with this type of Forer effect garbage, you'll be wasting everyone's time:

"You have a need for other people to like and admire you, and yet you tend to be critical of yourself. While you have some personality weaknesses you are generally able to compensate for them."

 

[Kuko 4000]

Perhaps you should get a sample profile that he has completed. If it's filled with this type of Forer effect garbage, you'll be wasting everyone's time:

"You have a need for other people to like and admire you, and yet you tend to be critical of yourself. While you have some personality weaknesses you are generally able to compensate for them."

Most of the examples he told me were very different from this, and I'm only going to include exact claims in the profiles.

Joe, you are correct about the astrology jargon and in the very beginning we agreed to left it out of the profiles. It will include only claims, not the astrological reasons behind the claims.

 

[SumDood]

I agree that giving the profiles directly to the test subjects might not be the best method. May I suggest a detailed questionnaire be given instead? Each claim can be represented in one of the questions and then the results can be tallied. For example, if the claim is "You have been divorced 3 times" you ask "how many times have you been divorced?" on the questionnaire. Of course this would only work if all the claims are fairly specific. Really not that different from what you have suggested, but it takes much of the analysis away from the test subjects. By doing so you would reduce the results to a more statistical analysis.

 

[JoeTheJuggler]

But then the results depend on the accuracy of the personality profile, which has its own issues.

True, but there is validity data on the various instruments. While it's not the best, it's a lot more objective than what's being proposed. And at least what I suggest puts the matching and "hit" judgment in the hands of the astrologer. The protocol proposed, for example, could be sabotaged by uncooperative skeptical subjects. (Before someone claims the same problem could be encountered with a skeptical subject purposely fudging his answers on the personality profile, these things are designed with an eye to thwart such attempts--by asking the same sort of thing multiple different ways to consider how consistent answers are.)

I agree it's not the perfect solution, but it makes a lot more sense to me than what was suggested. Plus it would be much simpler to do logistically. There might even be test result and birth info data already available (so that we needn't be concerned with skeptics attempting to bias the results).

If the claim of the astrologer is that a person's personality is determined or influenced by the apparent positions of the planets at his birth, it seems that trying to match up birth info with objectively derived personality profiles is the most direct way of doing it.

 

[JoeTheJuggler]

I agree that giving the profiles directly to the test subjects might not be the best method. May I suggest a detailed questionnaire be given instead? Each claim can be represented in one of the questions and then the results can be tallied. For example, if the claim is "You have been divorced 3 times" you ask "how many times have you been divorced?" on the questionnaire. Of course this would only work if all the claims are fairly specific. Really not that different from what you have suggested, but it takes much of the analysis away from the test subjects. By doing so you would reduce the results to a more statistical analysis.

I like that idea too, but I would be in favor of using existing questionnaires since there is already validity and reliability information available on them.

As I mentioned, it would be ideal if someone could find a database of already recorded test results (and the pertinent birth information).

ETA: Or perhaps tighten up the claim of the astrologer. Could he discern the person's gender from just the birth information? Or marital status? (The marital status one, though, would be pretty easy for the astrologer to get a statistical advantage just based on age.) Maybe number of siblings? Or a very simple description of the person's current occupation? (Toss out those that have age-related give-aways such as "student".) Maybe political party affiliation? (Though birth location might give a strong hint on that.)

 

[JoeTheJuggler]

Joe, you are correct about the astrology jargon and in the very beginning we agreed to left it out of the profiles. It will include only claims, not the astrological reasons behind the claims.

Sorry--I came in late and didn't review the first page of the thread. Still, I'm not sure it's a simple matter to eliminate everything that might be a hint to known sun-signs.

If possible, I'd prefer to see that avenue of information leakage removed altogether.

 

[Hokulele]

Sorry--I came in late and didn't review the first page of the thread. Still, I'm not sure it's a simple matter to eliminate everything that might be a hint to known sun-signs.

If possible, I'd prefer to see that avenue of information leakage removed altogether.

I believe that the astrologer in question is using Vedic charts, so for the most part, the sun sign hint issue is moot.

Also, I don't think this was meant to be a serious test in any sense, but more a chance to show the astrologer how his own confirmation bias has been working (as opposed to his clients' confirmation bias). From what Kuko has written, I believe a combination of warm and cold reading techniques have been responsible for the majority of this particular astrologer's hits, so this test would at best demonstrate to the astrologer that he isn't as successful as he thinks he has been.

In other words, this test is purely for the astrologer's sake, not for any of the watching skeptics.

Of course, if the astrologer does do something spectacular here, I would very interested in setting up something a lot more stringent and seeing how far this can go.

 

[Kuko 4000]

In other words, this test is purely for the astrologer's sake, not for any of the watching skeptics.

Of course, if the astrologer does do something spectacular here, I would very interested in setting up something a lot more stringent and seeing how far this can go.

Well, you are correct, but he is already asking if it's ok for him to use the test results publically...so I would like this to be stringent. I'm sure he would trumpet the results all over the world if he would PASS the test. This is why I've not proceeded with the test yet, it's way too muddy at the moment for me.

If you or anyone else here can think of a better protocol than this that requires 4 profiles max from the astrologer I'd more than welcome it! I'm willing to do the work.

 

[Hokulele]

Ah, if he wants to use the results publically, then that changes everything.

I would recommend using the questionnaire method mentioned earlier. Try to find people as close together in age as possible. Ask very specific questions that have to do with personality, but about specific events rather than traits. For example,

- Have you ever been married?
- Have you ever been divorced?
- How many children do you have?
- Are you self-employed?
- Do you own your own house/apartment?
- Are you currently living with someone else?
- Is that someone else a spouse, partner, or roommate?
- Do you have more than 6 months income in a bank savings account?
- Do you have retirement savings?
- What is your favorite ice cream flavor?
- Do you have any allergies?
- Do you own a pet?
- If so, what type of animal?
- Do you participate in organized sports?
- Do you have a college degree?
- If so, what was your major?

And so on. Depending on the number of questions, you could crunch the numbers to see how many hits are required to be a "match".

 

[JoeTheJuggler]

I believe that the astrologer in question is using Vedic charts, so for the most part, the sun sign hint issue is moot.

No, I mean he could take advantage of the fact that most people know their own sun signs and what things are typically associated with it.

In other words, this test is purely for the astrologer's sake, not for any of the watching skeptics.

Good point. In other words, there's no need to rule out cheating.

If that's so, then why not really simplify it. Have all the volunteers write up a description of themselves (their personalities, their careers, their families) just leaving out references to their age and geographical location. Then see if the astrologer can match up those descriptions with the birth info.

ETA: I'm basically suggesting the same thing as the questionnaire except putting it in prose form. I think you, Kuko, as the test administrator could strip out or censor any give-aways. I still say using a standard questionnaire (personality profile) that has years of data on exactly how reliable and valid it is would be preferable in a more strict protocol--and again, preferable with existing data rather than people testing for this purpose.

 

[Hokulele]

No, I mean he could take advantage of the fact that most people know their own sun signs and what things are typically associated with it.

My point was that the Vedic charts do not use the sun signs the way Western astrology does. For example, references to being "stubborn as a bull" in Vedic astrology does not mean you are a Taurus in the Western system. In fact, references to the zodiac could very well work against the astrologer in this type of test, where most of the participants ae used to Western categories.

Good point. In other words, there's no need to rule out cheating.

If that's so, then why not really simplify it. Have all the volunteers write up a description of themselves (their personalities, their careers, their families) just leaving out references to their age and geographical location. Then see if the astrologer can match up those descriptions with the birth info.

I would have agreed with you earlier, but since Kuko revealed that the astrologer is trying to use this as public proof, cheating has become a major problem. Originally I thought this was similar to the stories told by people like chillzero and Miss Anthrope, people who truly believed they had a gift and were sincere about demonstrating their "powers". When faced with the hard reality that cold reading can account for most of what most psychics and astrologers accomplish, I have seen people admit that they aren't gifted after all (chillzero's and Miss Anthrope's stories weren't exactly like this, but the general principle applies).

The fact that the astrologer is planning to use this test as promotional materials suggests that there is a willful ignorance, or even intentional deception, involved.

ETA: I'm basically suggesting the same thing as the questionnaire except putting it in prose form. I think you, Kuko, as the test administrator could strip out or censor any give-aways. I still say using a standard questionnaire (personality profile) that has years of data on exactly how reliable and valid it is would be preferable in a more strict protocol--and again, preferable with existing data rather than people testing for this purpose.

After the most recent revelations, I basically agree. The only problem with the better personality tests is that they can be more work than the average volunteer may want to do. It may be better to focus the test even more down to something very specific. Rather than casting the entire chart, it may be better to focus on answers for all 10 volunteers to a narrow range of personality traits. For example, get the CV's or resumes of each of the 10 volunteers and strip out years and locations, just leave the basic job descriptions and lists of responsibilities. Have the astrologer match the resume to the birth date/location.

If the volunteers are all roughly in the same age range, not only the job types but the number of different jobs held should be indicated in their charts.

Of course, assuming astrology isn't bunk.

 

[Tapio]

The fact that the astrologer is planning to use this test as promotional materials suggests that there is a willful ignorance, or even intentional deception, involved.

I also know the guy you're talking about. I admit everything might be possible, but I have to say I'd be absolutely shocked if I found out he's trying to intentionally deceive someone. Definitely not the type.

I believe this guy to be the perfect example of a person who simply hasn't crirically evaluated the way he perceives the world. Not once. Or if he has, the pain of admitting to his own insecurities has just been too overwhelming and so he's had to shut down the voice of reason before it even begun to rise.