Or, we can say that for all practical purposes, 3 of the others are so close to being insignificant that considering the subjective nature of the data, the incompleteness of the data (some large protions of properties unknown) the test can't confirm nor deny that the KNOWNS are primarily UNKNOWNS.
Let me first define a couple of terms:
The chance of a Type I error is the chance of rejecting the null hypothesis when it is true. The null hypothesis is that there is no difference between the groups. For all practical (real world) purposes there is no difference between a 5% chance of a Type I error and a 6% chance of a Type I error.
Regarding the first analyses:
COLOUR: | χ2 = 26.15| (df=8)| p<0.001 cutoff = 26.124
NUMBER: | χ2 = 40.73| (df=4)| p<0.001 cutoff = 18.467
SHAPE: | χ2 = 29.05| (df=6)| p<0.001 cutoff = 22.458
DURATION: | χ2 = 49.49| (df=7)| p<0.001 cutoff = 24.322
SPEED: | χ2 = 37.93| (df=5)| p<0.001 cutoff = 20.515
BRIGHTNESS: | χ2 = 5.85| (df=5)| p<0.05 cutoff = 11.070
We can see that the chance of rejecting the null hypothesis when it is true is
insignificant (like a 0.1% chance!) in all but the Brightness category. That is, there is less than one tenth of one percent chance of being incorrect in concluding there to be a significant difference.
For the Brightness category there is (approximately) a 30% (or more) chance of making a Type I error (considering that p<0.20 cutoff=7.289 and p<0.975 cutoff=0.831). That is, the chances are
very substantial that if we conclude there
is a significant difference here - we
will be committing a type I error.
Now let us see what happens when we remove the
astronomical KNOWNS from the analyses.
COLOUR: | χ2 = 13.40| (df=7)| p<0.05 cutoff = 14.067
NUMBER: | χ2 = 23.73| (df=4)| p<0.001 cutoff = 18.467
SHAPE: | χ2 = 11.46| (df=5)| p<0.05 cutoff = 11.070
DURATION: | χ2 = 14.43| (df=7)| p<0.05 cutoff = 14.067
SPEED: | χ2 = 43.71| (df=4)| p<0.001 cutoff = 18.467
BRIGHTNESS: | χ2 = 10.68| (df=4)| p<0.05 cutoff = 9.488
Now we can see that Colour, Shape, Duration and Brightness are all hovering at about a 5% chance of making a Type I error. In strictly
statistical terms, a 5% chance is considered to be an acceptably small enough chance of committing a Type I error to allow a positive conclusion (in this case that there IS a
real world difference between the KNOWNS and the UNKNOWNS). In
practical (or real world) terms, the difference between a 6%, a 5% and a 4% chance of committing a Type I error is negligible.
Taking Colour as an example we can see that the 5% cutoff is 14.067. The test statistic actually turns out to be 13.40. This would relate to about a 6% chance of making a Type I error (if we were to conclude there is a difference between the KNOWNS and the UNKNOWNS). Now the
statistical cutoff is 5%, but in the
real world a 6% chance is just as acceptable. As Jacob Cohen (a giant in the world of statistics) points out in his seminal paper [“The Earth is Round (p < .05)”]
“After 4 decades of severe criticism, the ritual of null hypothesis significance testing – mechanical dichotomous decisions around the sacred .05 criterion – still persists” (Cohen, J. (1994) The Earth is Round (p < .05). American Psychologist, Vol. 49, No. 12, 997-1003)
In the real world, what this all means is that
there IS a difference between the KNOWNS and the UNKOWNS. So when you state
” the test can't confirm nor deny that the KNOWNS are primarily UNKNOWNS.” you (and the authors of the report) are simply incorrect. The tests DO confirm a difference. The bottom line is that the Chi_Square analyses
confirm that there IS a difference between the KNOWNS and the UNKOWNS. In the first analysis it was all the characteristics except Brightness (which we truly can say on this test there was
not a significant difference). The second analysis confirms a practical difference on all the characteristics examined.
No matter how you spin it Jocce, that is the bottom line here. There IS a difference between the KNOWNS and the UNKNOWNS.
With regard to your “subjectivity of data” statement we must note that
the more reliable the report, the more likely it was that it would be categorised as UNKNOWN (and a full 38% of the
most reliable reports – those from trained military observers – were categorised as UNKNOWN!).
That is
another bottom line here Jocce. No matter how you spin it, that fact remains.
As to your “incompleteness of the data (properties unknown)” statement, we must note that the data were NOT incomplete. First, the analysts had a special INSUFFICIENT INFORMATION category into which they placed all such reports as they considered to be of that kind - and then
excluded those from further analysis. Second, during the Chi-Square analyses, they also
accounted for reports which did not specify the particular characteristic under analysis. There simply was
no incomplete data that could have affected the results.
That is yet
another bottom line here Jocce. No matter how you spin it, that fact remains.
Regarding your statement about the reports being filled in “a long time after the event”, the report states:
"most of the reports were not reduced to written form immediately. The time between sighting and report varied from one day to several years." (p. 3)
I have already noted that there is no data provided as to the distribution of the length of time between sighting and report. Given the author’s propensity to misrepresent the data and the results of analysis (if not outright lie about them) I would not be surprised to find out that there was but a
single report where the delay was “several years”. Indeed, it is highly unlikely that in their analysis of the reports, that such a delay in reporting would not have been factored into the reliability analysis. And then Jocce, if we
exclude the least reliable reports, the percentage of UNKNOWNS becomes
greater!
Yet another bottom line here Jocce. No matter how you spin it, that fact remains.
With regard to your question about the statistical significance of the
trend in the data (that the more reliable the report, the greater the percentage of UNKNOWNS), I noted that such a question was literally senseless. That is, it makes no sense in any statistical or practical terms to ask such a question. It is a dichotomous relationship. You either
include the less reliable reports, in which case the percentage of UNKNOWNS drops (to a minimum of 21.5%). Or you
exclude the less reliable reports, in which case the percentage of UNKNOWNS increases (to a maximum of 38%).
Still
another bottom line here Jocce. No matter how you spin it, that fact remains.
Jocce, in the face of your statements that clearly reflect either a misinterpretation or a misunderstanding of the data (and the results of analysis) I will simply present the FACTS. That is another bottom line here Jocce.