Bwahahaha! You must have been sick the day they taught multivariate analysis in statistics. With a wave of your hand, you simply disregard an entire field of statistics, and one of the most important techniques in experimental research that guarantees the independence of the one variable you were give data for.
What's even worse is that I described in detail how they are used in research of the type PEAR conducted, and incidentally in medical research. They are invaluable in ensuring the integrity and applicability of the study, and I took great effort to explain why. Do you address my detailed description at all? No, you simply announce that I'm wrong and give a highly simplified description of the one part of some clinical trial you may have been incidentally involved with, and then wrongly concluding that's all the statistical work that was ever done or needed to be done.
You keep proving time and again that you are unwilling to expand your knowledge to fit the problem. You keep trying to pare away the parts of problems that don't fit your existing understanding. And hubristically you try to call other people stupid when they point out your oversimplifications and errors. What kind of a "consultant" does that?
But you're not an authority on what is "usually" done in experiments. In the thread where you gave us empirical anecdotal evidence for reincarnation, you both illustrated and admitted that you didn't know what empirical controls were. Do you really expect us to forget about these things from day to day?
A subject pool is first established, usually from volunteers out of the general human population. This is true both for physiology research and for psychology research. A common source of psychology research subjects is undergraduate students, who must participate in order to get credit in the psychology classes they take. From this subject pool are drawn the subjects for some particular study based on their demographic conformance to the general population, or to the parameters of the study -- sometimes both. Usually the results are hoped to generalize to the population at large, so the sample is desired to differ as little as possible from what is known about the general population. Often the study is limited to subjects of a certain category, such as to women who have had at least one child. Obviously the men and childless women must be filtered out in that case. Sex is a category. Number of children is a category. The latter especially may be one of the proposed correlates.
Drawing at random often works because subjects often naturally fall into a representative (or desired) demographic distribution naturally. A random sample of a representative pool would also be representative to within a certain amount. Another method is to score each member of the subject pool according to how closely he fits the general population (or study parameters) and then take only N of the top scorers.
Randomness here is not the end goal. The end goal is conformance to the parameters of the study. Randomness is only a proxy for that, and not even always the best proxy.
So with an appropriately shaped subject sample in hand, what happens next? Well, as I explained previously, it's separated into the control and variable groups. In a clinical trial, the control group gets the placebo. In a psychology experiment there may be no control group, the sample having been homogenized against the general population. Where the bifurcation occurs, it is best done randomly to avoid experimenter bias. However it's done, the goal is to ensure that all the pertinent categorical variables are equally distributed on both sides of the divide. This is to ensure that the two groups are otherwise independent, besides the thing you're going to do to one group but not the other.
This is measured by looking at the pertinent categorical variables and determining whether the values those variables take on is related to which group they were randomly placed it. Here too, randomness per se is not the end goal. The end goal is homogeneity between the two experiment groups. Randomization is merely one way to achieve it.
Regardless of how it's achieved, it will never be perfect. Neither randomization process produces a sample that's perfectly aligned with the population from which it's drawn, nor a pair of groups that's exactly equally divided among potential confounds. It just avoids a particular sort of bias. But the groups will always differ a measurable amount in all the ways the matter for the experiment, and the sample as a whole will always differ a measurable amount from the population they are meant to represent.
The point here is that we can measure it. This is what statistics, at its heart, is all about. We'll come back to this.
Total hogwash.
Data are retrospectively excluded from studies all the time for reasons such as measurement error, subjects' failure to follow the protocol, experimenter error, the death or withdrawal of the subject, and so forth. In any sufficiently large study it is practically unheard of for all the subject data to make it to the end of the study. Now perhaps in the data you were given, those inappropriate data had already been culled. Once again you continue to misunderstand why your particular suggestion for removing data was invalid. Because you were challenged (correctly) on that particular reason, you seem to have evolved this childlike belief that no data may ever be excluded from analysis. If your concept of science were true, think of how many studies would have to be terminated, after years of work and funding, because one data point became unusable for whatever reason. No science would ever get done.
Emphatically no, and it illustrates just how little you really understand what statistics are for. Randomness is not itself the desired end. It is the means to other ends, and those ends are not disserved by the departure of one subject's data after the randomization process has done its thing, nor are they incorrigible. Statistics is about confident reasoning in the face of uncertainty. Your view is that a proper scientific study is a perfectly balanced house of cards that can't tolerate the merest jiggle of the table without crashing down in its entirety. That's as anti-statistical a sentiment as there ever could be. Statistics is about what you can do with what you know, even if it's not perfect.
Those subjects were never perfectly aligned anyway, randomness notwithstanding. There was always a need to measure how aligned they were to the population and to each other and to develop a confidence interval to describe how much those groups' behavior could be considered useful. When one subject departs, the confidence interval changes. But it doesn't just go up in smoke. That's what it means to reason statistically. Now if your N is too small, then one subject gone may change the confidence interval to the point where significance of effect is hard to measure. But that's not even remotely the same thing as the naive all-or-nothing fantasy you've painted.
You told me I hadn't provided an example of these things. That's a lie, because I've referred twice now to Dr. Philip Zimbardo and one of the most famous psychology experiments of all time, the Stanford prison experiment. You may recall this was formulated as an experiment to determine the effect of imposed roles on cruel behavior, and is most infamous for having developed astonishing levels of brutality before it was belatedly terminated. In his most recent book, The Lucifer Effect, he gives a retrospective of the experiment interleaved with copious mea culpa for not realizing what was going on. Naturally it's all in the original paper, which is easily available. But in Lucifer he takes you through it in detail, assuming you aren't a research psychologist.
So the first thing Zimbardo had to do was get reasonably suitable subjects. He described his recruiting efforts; most ended up being undergraduates at Stanford University, for obvious reasons. They weren't randomly sampled from the entire U.S. population, but they reasonably varied in demographics as much as college students can be expected to vary.
Then he administered a set of standard psychometric instruments to them to determine where they fell according to various variables he knew or suspected would lead to brutal behavior. He describes the variables and shows how his sample scored as a group along all these dimensions. He also compared their scores with distributions of scores from a the wider population. It's obvious that Zimbardo's sample was not perfectly representative of the U.S. population in all respects. For starters there were no girls, owing to the practical limits of human-subjects testing on this particular experiment. But it doesn't largely matter because he merely needed to establish by psychometry that they were representative enough in the ways that applied to his study.
Then he randomly divided the sample into Guards and Prisoners and showed how the psychometric evaluations were distributed between the two groups. Specifically he showed that in each of those categories the distribution of the psychometric variables was so similar between the two groups that they could be considered equivalent in terms of their predisposition to violence and brutality. Not identical, of course. There was variance. But one of the things the t-test sort of analysis can give you is a comparison between two population means, knowing the confidence interval that applies. While the groups were not identical, the confidence interval was broad enough that it could contain the (unknown) theoretical mean score for all the subjects. This is what we mean by homogenization. Unless you do it, you can't independently attribute effects to the imposed cause.
Zimbardo planned to impose different roles on the two groups, and he needed them to be otherwise as similarly as possible, so that any differences he observed could be properly attributed to the imposed role, not the emergence of some latent property that was skewed in one group versus the other. Hypothetically, if all the people with measured aggressive tendencies were somehow in the Guard group, it would be hard to chalk up a beat-down to the imposed role and not the predetermination. Or in contrast, if all the non-white people were in the Prisoner group, you couldn't separate the Guard-Prisoner dichotomy from the race category and say that everyone's behavior would be the same even if all the prisoners had been white.
All the things I've been talking about are in the book.
The rest of the experiment is history. Zimbardo let it run far too long, up to and past the point of inflicting severe emotional distress on the participants. He spent quite a lot of time cleaning up the mess. But what has emerged over time is that he has maintained contact -- and even professional relationships (some of them became psychologists themselves) -- with the former subjects. And it emerged that one member of the Guard group developed his own agenda that was not necessarily compatible with the goals and protocols of the experiment. For the quantitative part of the experiment Zimbardo had to decide whether to include that data.
A simpler example suffices. Suppose there is a drug trial and subjects are prohibited from drinking alcohol for the duration of the study. Now let's suppose the experimenter discovers that one subject is having his daily glass of wine at dinner, in contravention of the protocol. According to you, we would have to include his data in the final result even though he broke the experimental protocol and confounded whatever results he may have had from the drug with the result of drinking alcohol. That's what would affect the study, not booting him out. If we boot him out of the study, whichever group he was in -- placebo or drug -- has its N-value decremented by one, and his contribution to the homogenization check is removed. And for all we know, his departure might make the groups more homogeneous. This retains the integrity of the study because the remaining subjects (who are presumed to follow the protocol) are exhibiting the behavior it was desired to study.
No, that's not even remotely close to what Palmer suggests.
No. As explained above, randomization is the means we use to achieve various ends which are then not affected by culling bad data in a way that statistical analysis cannot overcome. Randomization is not an end unto itself. This is your pidgin-level knowledge of statistics and experimentation, against which you're trying to measure the behavior of the published experts. You're literally trying to tell career scientists "they should have known better," when you have already both demonstrated and admitted you don't know their science or its methods.
