Could you elaborate on this? What is the problem with Bem's DR%?
Sure.
In his retroactive priming experiments, Bem was attempting to measure differences in recall between two conditions: practiced (P) and control (C). To assess the difference, he computed the following statistic for each subject in the study:
DR% = [(P – C) × (P + C)] × (100% / 576)
In this statistic, the term 100%/576 is just a scaling factor designed to convert the scores into convenient units. Since it is just a constant that each subject's score is multiplied by, it has no effect on the analysis, and we can ignore it, concentrating instead on the term that matters: (P – C) × (P + C). This term is the raw difference in recalls (P – C) multiplied by the total number of correct recalls (P + C). Bem characterized the factor (P + C) as a weighting factor designed to give more weight to subjects who recalled more words in total.
My objections to Bem's DR% are fourfold:
1. Bem's DR% is a nonsensical measure of differential recall; that is, it's wrong in theory.
2. Bem's DR% is an erroneous measure of differential recall. It results in actual errors in the analysis. That is, it's wrong in fact.
3. Bem's rationale for weighting the data by the total number of words recalled is wrong in principle.
4. The resulting weights are wrong, in fact.
I'll address each of these objections in turn.
1. DR% is a nonsensical measure of differential recall
It doesn't take much mathematical muscle to show the following:
(P – C) × (P + C) = P2 – C2
What this shows is that Bem is using the difference between the square of the number of recalled practiced words and the square of the number of recalled control words to measure the effect of psi. Now, under what possible theory of anything would one think that the effect of psi would be on the square of the number of recalled words under each condition, rather than the much more reasonable choice of P – C itself? It's nonsense.
2. DR% is an erroneous measure of differential recall
Aside from being nonsensical, the use of DR% causes real errors in the measurement of differential recall. Whatever statistic is used to measure a subject's differential recall, it should accurately reflect the probability that that subject's observed difference could be due to chance; larger values of the statistic should consistently correspond to smaller probabilities that chance produced the result. DR% does not perform well on this criterion.
As an illustration, consider two subjects: Subject A recalls 18 practiced words (75%) and 13 control words (54%). According to Fisher's exact test, the gold standard for this kind of comparison, this difference cannot be distinguished from chance (p-value = 0.227). Subject B recalls 9 practice words (37.5%) and 0 control words (0%). This subject's result is highly statistically significant (Fisher p-value = 0.0016), a result inconsistent with mere chance. Now, let's compute each subject's DR% and compare them with the p-values for their results:
Subject A: DR% = 27; p-value = 0.227.
Subject B: DR% = 14; p-value = 0.0016.
In spite of the fact that Subject A's results were indistinguishable from chance, and Subject B's results were highly statistically significant, Subject A's DR% is nearly double Subject B's. Bem's DR%, thus, mis-ranks these subjects in terms the purported psi effect.
Below are the results of a computer simulation of 100 subjects that I ran, in which each subject was assigned two random integers, each ranging from 0 to 24, representing their correct number of recalls out of 24 practiced and 24 control trials. From the simulation results, I computed the following for each subject: absolute DR%, absolute percentage difference (| P/24 – C/24 |), and their Fisher p-value. The results are displayed in the following figure. In the figure, the vertical axis is the p-value (presented on a log scale for ease of readability). The horizontal axis represents the absolute values of DR% (red dots) and percentage difference (black dots).
The figure clearly shows that the correlation between the p-value and the absolute percentage difference is much stronger than the correlation between the p-value and the absolute DR%. The ranking of the subjects (measured by Spearman's rho), is nearly perfect for the percentage difference (rho = –.98 [–1 would represent perfect ranking]). For DR%, rho = –.86. Roughly speaking, in this context, the rho
2 for a measure of differential recall quantifies how well it predicts the correct ranking of subjects in terms of the p-values of their results. DR%'s performance is mediocre, with a rho
2 of 0.74 (vs. 0.96 for the percentage difference).
3. Bem's rationale for weighting the data is wrong in principle
Bem's rationale for the DR% was that each subject's data should be weighted by the total number of words they recalled. But why? Just saying they should be doesn't make it so. What Bem is doing is giving subjects with better memory greater weight in the analysis. But, if anything, this is backwards. If anything, memory would confound or modify the effect of psi; you'd therefore want to control for memory in the analysis, not amplify its effect.
4. The resulting weights are wrong
The data from this experiment are binary data (a word is either recalled or it is not) grouped by subject. Traditionally, there have been two ways to analyze this type of data, neither optimal: The first was to ignore the grouping and analyze the raw data. The problem with this approach is that it ignores the fact that multiple data points contributed by a subject contain less information than would the same number of data points contributed by different, independent subjects. The statistical power of the study is thus reduced. Since the power of a study is related to its sample size, when
n subjects contribute
p data points each, the effective sample size of the study is less than
np. Ignoring the grouped nature of the data treats the study as if its effective sample size were
np, exaggerating the statistical power of the study and the significance of the computed effects.
The second approach was to aggregate the data by subject, and analyze the aggregated data. However, this reduces the sample size to the number of subjects, excessively diminishing the statistical power of the study.
The truth of the matter is that the effective sample size will tend to be somewhere between the above extremes, depending on how highly correlated within subjects the data actually are. This can't be determined a priori. A rigorous statistical analysis would quantify the correlation and properly account for it in the results, effectively weighting the data based on the magnitude of the correlation. Statistical techniques to perform this type of analysis have been available for several decades, and are
de rigueur in many fields, though apparently not in experimental psychology. This is how the Bem data should have been weighted, not by the total number of words recalled by each subject.
I should point out that, in spite of the lengthy explanation above, the problems with Bem's DR% are minor in light of the deeper flaws that pervade the paper and the fundamentally misguided approach to research that produced them. I like the critique by
Wagenmakers et al (.pdf), as well as the prescient warnings of Kerr.¹
¹Kerr, N. L. (1998). HARKing (Hypothesizing After the Results are Known). Personality and Social Psychology Review, 2, 196-217.
