Merged Naive empiricism vs skeptical empiricism

[jt512]

At any rate, your argument demands that we dismiss Dibblee's work--and more, that Dibblee's work was unfit for publication at the time in which it was published.

If you think that, then you have completely misunderstood my argument. Indeed, it is obvious that you have completely missed my point, which is this: if we attempted to replicate every article in medicine published within the last month, then less than half would likely replicate. Do you get it now? This has nothing to do with advancements in medicine, or "paradigm shifts," or whatever. The validity of a clinical trial does not depend on any "paradigms." It depends on whether it is really true that treatment A cures more patients than treatment B or not. Something on the order of half of all medical papers are wrong. That means that if you take a randomly selected medical paper, it is basically a coin flip as to whether the findings are correct or not. In experimental psychology, the situation is probably even worse, with less than half (maybe way less than half) of findings being true.

If geology is in better shape than medicine or psychology than that is great for geology. But there are large swaths of science that are in serious trouble due to systemic errors in how their experiments are conducted, analyzed, reviewed, and published.

 

[lenny]

The fact that most science needs to be reproduced to be considered valid, but replication isn't usually attempted or published, shows that the process is broken.

i think there is something too this. the process certainly is not running smoothly, if not worse.

there is rarely a need for detailed replication: when things are advancing we build on what others are doing. when what foundation fails, well when what we build fails (and after we are sure the failure is not our new bits), then we go looking for what is not sound, and then the inability to reproduce someone else work leads to discussion. traditionally there was a heavy penalty for having wasted other peoples time.

the fact that often what is published cannot be build upon, that publishing four papers each with a quarter of an idea which is only coherently presented in the fifth paper... when this is seen to be more effective professionally than simply publishing only the fifth paper, then the process is overheating towards meltdown.

 

[lenny]

They've given up their priests and expect scientists to provide the same function. Ask a question, get the answer, no strenuous thinking required.

to what extent have (some) scientists actively stepped up to fill that role? some even attacking the priests with the aim of taking that role, claiming to offer a different but brighter light?

 

[lenny]

so who's fault is that? the public's or our's?

The fault lies with the public. I know of no journal that claims to provide The Truth. They only claim to provide the most recent data.

i think the "only" in your last sentence will prove difficult to defend. high quality journals, say the proceedings of the national academy of sciences in the US, often intentionally weigh in on matter of policy. explicitly.

that suggests more than claims of "best available"; it suggests "adequate for purpose."

and who is responsible for the explosion in "number of papers" per author? (or number of citations, or H index). attempts to shortcut to evaluation of a scientist's work result in gumming up the works. to the detriment of the advance of science.

Anyone who uses the number of publications as a criteria for evaluating any scientist, rather than the quality thereof.

agreed. and that tends to be "us", not the public.

one of the published feynman story's includes a nice quote he wrote when asked to provide a letter supporting a tenure application. a combination of "well he is just two offices from you, why don't you just go talk to him" and "what do you think of his paper on X/read his papers."

i think science would benefit on several levels if we could incentivise the young to write fewer, better papers. even then i do not have a good way to reduce 200 applications to a shortlist where those good papers could be read, but we'd benefit from a better way of evaluating young scientists.

 

[Dinwar]

This gets back to the topic of the other thread. Few scientists understand the issue.

Do you seriously not get how telling scientists "You don't understand what you're doing" is attacking us? Here's a bit of advice: if you're not a member of the field (and I have little doubt that you're an outsider looking in), you don't get to tell us how we operate.

We DO get the issue. What YOU fail to understand is the myriad of OTHER issues associated with this topic. Science isn't cheap, and no one--NO ONE--pays for replication. You want to insist that every experiment is repeated, go ahead and pay for it. Until you do--and I mean you, yourself, personally--do not expect us to bow to your whims.

Secondly, not all experiments CAN be repeated--at least, not exactly. Paleontological excavations are experiments. Astronomical observations are experiments. Biological expeditions are experiments. The class of experiments is termed "natural experiments". Often, the circumstances involved are unique; any individual aspect may be reproducible, but reproducing the exact experiment is simply not possible. You tell me how you're going to replicate a supernova. Or an impact between asteroids and a planet. Or a chance encounter with a new species of worm.

Furthermore, the demand that we replicate experiments before accepting their results flies in the face of logic. I invite you to Google Strong Inference and read the wiki page or the PDF that comes up to gain an understanding of why. Your idea has been considered. It's been rejected, for numerous reasons that you haven't given due consideration.

Your first sentence above is false on its face. If the process worked, then few published studies would be false.

This demonstrates why I doubt you're a scientist. Every scientist knows that there are all sorts of reasons why a published study may be false, even if we leave aside fraud, incompetence, and other such errors. Simply having an unknown bias--and many are unknowable at the time of publication--is sufficient to yield false results. Publish or Perish adds sufficient pressure to make poor studies go into print. Then there are controversies. One side is usually more wrong than the other, and typically all sides are incorrect to some degree--but we don't know which is which yet. Is space stringy or loopy? We don't know; we know one is wrong, but we don't know which. That means that false data is getting published. And some researcher somewhere may be doing an experiment that disproves your idea while your paper is in review. Happens more than you think.

Peer review is not a substitute for rational evaluation of the report, which is what your proposal boils down to. It cannot be. This worship of the peer review process that you're perpetuating is actually damaging to educating people about science, because it's an easy out when abused in the manner you are doing. Peer review means that the paper is good enough. It's up to those of us with our boots on the ground, so to speak, to determine if the paper is right. It's always been that way, and there's no way around it.

That is utter nonsense. Of course the peer review process evaluates the validity of the claims made in the article. That's the main point of peer review.

When you actually publish something, you'll learn your error. I'm not discussing hypotheticals here; I've been through the process. Yes, the reviewers often comment on the validity of certain arguments--but they DO NOT determine whether the paper is true or not. They CAN'T; for one thing, reviewers are professional scientists and don't have that kind of time. For another, new data can always overthrow even our most cherished ideas.

i think the "only" in your last sentence will prove difficult to defend.

It's trivially obvious. They can ONLY provide the most recent data (and the evaluation thereof). They can't provide data more recent than the most recent data; time travel doesn't exist.

agreed. and that tends to be "us", not the public.

It's both. And I've always detested such metrics as impact factor and publication number. I've spoken out against them several times on this board and on more professional email lists.

 

[lenny]

When you actually publish something, you'll learn your error. I'm not discussing hypotheticals here; I've been through the process. Yes, the reviewers often comment on the validity of certain arguments--but they DO NOT determine whether the paper is true or not. They CAN'T; for one thing, reviewers are professional scientists and don't have that kind of time. For another, new data can always overthrow even our most cherished ideas.

it seems to me that you are trying to generalize a bit too widely from a class of papers that are of interest to you.

they DO NOT determine whether the paper is true or not. They CAN'T;

two cases where they can, and often do, are (i) errors in the mathematics and (ii) internal inconsistencies in the results.

 

[lenny]

It's trivially obvious. They can ONLY provide the most recent data (and the evaluation thereof). They can't provide data more recent than the most recent data; time travel doesn't exist.

there is more to science than data, and there is more to data than numbers. agreed?

of course it is true that scientist's cannot provide anything better than "best available"; nevertheless while science can never know that this will prove adequate (for the reasons you note), we can know that it will prove inadequate (without time travel).

 

[jt512]

Do you seriously not get how telling scientists "You don't understand what you're doing" is attacking us? Here's a bit of advice: if you're not a member of the field (and I have little doubt that you're an outsider looking in), you don't get to tell us how we operate.

I am a statistician who works closely with scientists. I know very well how science operates, at least in the fields I am familiar with, which, admittedly does not include geology, which I take it is your field. Indeed, in fields that are highly dependent on statistics, such as experimental psych, in some ways I understand certain crucial aspects of the methods that the field uses better than many practitioners of the field itself. Indeed, I'm in the process of writing a paper critical of certain practices in experimental psych.

We DO get the issue. What YOU fail to understand is the myriad of OTHER issues associated with this topic. Science isn't cheap, and no one--NO ONE--pays for replication.

Funding problems don't change the fact that unreplicated research findings should, in general, not be considered proven. If medicine and experimental psych don't do what needs to be done to change whatever faulty practices they are using, then they will continue to produce many (arguably, mostly) invalid results. Sorry, if you find this an inconvenient truth.

Secondly, not all experiments CAN be repeated--at least, not exactly.

You should try actually reading what I write. I have already said that replications don't have to be exact.

Furthermore, the demand that we replicate experiments before accepting their results flies in the face of logic. I invite you to Google Strong Inference and read the wiki page or the PDF that comes up to gain an understanding of why. Your idea has been considered. It's been rejected, for numerous reasons that you haven't given due consideration.

First of all, I am not "demanding" that anybody do anything. That would be silly. All I am doing is pointing out the fact that the scientific practices of some fields have resulted in those fields publishing a great deal of invalid results. One reason for this is their reliance on frequentist significance testing, which does not permit inferences to be made about the truth of a hypothesis from a single experiment alone. Frequentist significance testing requires replication. Other methods of statistical inference do not. Perhaps this "strong inference" you mention is one of them, but whatever that actually is, it has not been widely (or at all) adopted in any scientific field that I am familiar with.

Peer review is not a substitute for rational evaluation of the report, which is what your proposal boils down to.

Put words in other peoples' mouths much?

This worship of the peer review process that you're perpetuating is actually damaging to educating people about science...

Um, yeah, I guess you do. I hardly worship the peer-review process. I think it is badly broken, and I have said so.

Peer review means that the paper is good enough.

It doesn't even mean that. Arsenic bacteria, anyone?

When you actually publish something, you'll learn your error.

Um, I've published plenty, and continue to. But thanks. It's always fun to read condescending comments from ignoramuses. And by the way, I occasionally am asked to referee a paper, which, if I'm not mistaken, you have stated you are not. So, watch the unwarranted assumptions, eh?

I'm not discussing hypotheticals here; I've been through the process. Yes, the reviewers often comment on the validity of certain arguments--but they DO NOT determine whether the paper is true or not. They CAN'T; for one thing, reviewers are professional scientists and don't have that kind of time. For another, new data can always overthrow even our most cherished ideas.

Sorry, but you don't know what you're talking about. If a reviewer believes the conclusions are invalid, then they will recommend the paper be rejected or substantially revised. My girlfriend, who is on the editorial board of several scientific journals and reviews hundreds of papers a year, estimates that she has in the past (she seems to be sent better papers lately) recommended rejection of 20% of manuscripts because the authors' conclusions were not supported by the work they performed. For an example where the reviewers failed to do just that, see the aforementioned arsenic bacteria paper that should have been rejected, but was actually published in Science.

It's trivially obvious. They can ONLY provide the most recent data (and the evaluation thereof). They can't provide data more recent than the most recent data; time travel doesn't exist.

That is indeed trivially obvious, so you should stop writing it, as no one has suggested that it is anything other than trivially obvious.

 

[CapelDodger]

to what extent have (some) scientists actively stepped up to fill that role? some even attacking the priests with the aim of taking that role, claiming to offer a different but brighter light?

I really couldn't say.

Plenty of people have stepped up with ideologies to provide absolute answers to any question, and many people have turned to them for guidance (especially in troubled times). That's more of a problem to my mind.

 

[CapelDodger]

it seems to me that you are trying to generalize a bit too widely from a class of papers that are of interest to you.

It seems to me an awful lot is being said about medical trials, and then being generalised to science as a whole. I don't find this valid, as medical trials have their own peculiar problems (not least of which is lack of transparency).

two cases where they can, and often do, are (i) errors in the mathematics and (ii) internal inconsistencies in the results.

One can indeed determine that a paper is not valid, or that its conclusions are not justified.

 

[jt512]

One can indeed determine that a paper is not valid, or that its conclusions are not justified.

That's a good point, and possibly a source of the disagreement between Dinwar and me. He seems to think I expect reviewers to confirm the validity of authors' conclusions, when I mean that I expect them to reject papers that are invalid, that is, whose conclusions are not warranted.

 

[jt512]

Furthermore, the demand that we replicate experiments before accepting their results flies in the face of logic. I invite you to Google Strong Inference and read the wiki page or the PDF that comes up to gain an understanding of why. Your idea has been considered. It's been rejected, for numerous reasons that you haven't given due consideration.

So I read the original paper on "strong inference" by Platt. There is absolutely nothing in it that suggests that replication is unimportant. What on earth are you thinking?

On the contrary, if you are going to make major decisions on the direction of future research based on the result of a single "crucial experiment," then you had better be sure that the result of your "crucial experiment" is true, rather than just a fluke—that is to say, the result is replicable.

ETA: in an earlier post I erroneously wrote that my g/f reviews "hundreds" of papers a year. I asked her, and she says that actually she reviews about 50 papers a year, but declines requests to review another 50–100 papers per year.

 

[lenny]

Platt's paper is here.

 

[CapelDodger]

... f you are going to make major decisions on the direction of future research based on the result of a single "crucial experiment," then you had better be sure that the result of your "crucial experiment" is true, rather than just a fluke—that is to say, the result is replicable.

This would seem to be the class of papers we should be concerned about - those on which important decisions are based. They're also the ones most likely to be pored over, argued about, and indeed replicated (where feasible). In such cases, unpublished negative results by other parties are likly to be resurrected.

In the case of medical trials, replication by independent parties is often not feasible due to cost and the dread "commercial confidentiality", while negative results by the same parties are more likely to remain unpublished. That, I think, is a definite problem, but hardly one of Science per se.

 

[Dinwar]

This would seem to be the class of papers we should be concerned about - those on which important decisions are based. They're also the ones most likely to be pored over, argued about, and indeed replicated (where feasible). In such cases, unpublished negative results by other parties are likly to be resurrected.

It's also a complete misrepresentation of what happens. While the exact experiment is generally not ran multiple times, experiments that have implications for the idea being tested are often ran. Few ideas in science can only be tested in one way. This is why meta-analyses and compelation works in science are so valuable: they pull together disparate research into a coherent and relatively accessible whole.

A practical example is the "Taxonomy as a Rigorous Science" thread. The whale taxonomy serves as a test for the reliability of multiple means of reconstructing evolutionary history. We've done the direct experiments with various methods of reconstructing evolutionary history; however, every time we compare two or more (such as the genetic data vs. morphology data in that whale paper) it serves as yet another test.

Because all conclusions are tentative in science, and only held until something better comes along, it's perfectly acceptable to take an idea and run with it just to see how far you can get before it shatters. That's what a lot of science boils down to (essentially, this is what all science supporting the current paradigm is). Then, once the idea breaks down, we sort through the pieces and figure out how to make a better one.

I don't need to directly reproduce Gould and Eldredge's exact study to test punctuated equilibrium. It's perfectly acceptable for me to say "If PE is true, we should see X under conditions Y" and to look for them in a completely different taxa, in a completely different formation, at a completely different time. It's even acceptable for me to entirely miss the implications of my research as it partains to PE, and for someone else to say "Wait, his study of X taxa seem to fit with the PE mode better than any other evolutionary tempo", and for that to count as a test. Science isn't half as rigid as middle school textbooks portray it.

 

[CapelDodger]

A practical example is the "Taxonomy as a Rigorous Science" thread. The whale taxonomy serves as a test for the reliability of multiple means of reconstructing evolutionary history. We've done the direct experiments with various methods of reconstructing evolutionary history; however, every time we compare two or more (such as the genetic data vs. morphology data in that whale paper) it serves as yet another test.

We may have different perspectives, but I don't class this as something on which important decisions are based.

Because all conclusions are tentative in science, and only held until something better comes along, it's perfectly acceptable to take an idea and run with it just to see how far you can get before it shatters.

If running with it involves building a bigger LHC you'd better have your ducks in a row before pitching the idea, and be pretty damn' sure they really are ducks and not swans.

The point I'm trying to make is that Science isn't "broken" because there's a problem with medical trials - which I see as the generalisation being made in the Economist piece. (I also see that piece as part of an effort to "knock Science off its perch" but that may be a diversion too far.) If each field of research (such as yours) is examined on its own merits it becomes clear that there's no great problem.

 

[Dinwar]

If running with it involves building a bigger LHC you'd better have your ducks in a row before pitching the idea, and be pretty damn' sure they really are ducks and not swans.

Of course. I didn't mean to imply that science was frivolous. Rather, I meant that it's entirely justifiable to say "Okay, this appears to be true. If it is, this other idea is also true. Let's look for this other idea."

We do this all the time. For example, our search for gravity waves. We've no direct evidence for their existence (that I know of; I may be wrong here). However, they're a necessary consequence of other theories for which we DO have a great deal of evidence. So it's justifiable to look for such waves.

We still may be wrong. Such waves may not exist (again, going off the latest I've heard, which isn't very solid; but it suffices for the purpose of illustrating what I'm talking about). Future work may show that we missed some aspect of the universe that makes them impossible. This isn't a bad thing; this is why we run tests, instead of simply assuming that we're right, after all. Even the most well-established theories can be overthrown if the data contradict them.

Furthermore, I'm arguing that the search for gravitational waves counts as a test for those theories which predict them. It's not a direct test, to be sure, and we're certainly not replicating anything--but by testing the implications of an idea we are in fact testing the idea. If we fail to find gravitational waves we have to reconsider the concepts which lead us to conclude that they exist. What this means is that several posters in this thread are dead wrong about numerous points. Our ideas ARE repeatedly tested--just, no two tests are going to be identical.

The point I'm trying to make is that Science isn't "broken" because there's a problem with medical trials - which I see as the generalisation being made in the Economist piece.

No arguments from me on that point. I detest the concept of taking one subset of science and declaring that it's Science as such, regardless of which science it is or why the arguer is doing so. Of course, those of us in historical sciences are a bit overly sensitive to that sort of thing; we all have to go through a period of proving to ourselves that historical sciences are science, since they don't exactly follow The Scientific Method present in middle school.

 

[jt512]

For example, our search for gravity waves. We've no direct evidence for their existence (that I know of; I may be wrong here). However, they're a necessary consequence of other theories for which we DO have a great deal of evidence. So it's justifiable to look for such waves.

And how do you think physicists will test for these waves? By a single experiment? I doubt it. Physicists seem to prefer that results be replicated before claiming to have made a discovery. Consider the Higgs boson search at CERN:

On 4 July 2012 both of the CERN experiments announced they had independently made the same discovery: CMS of a previously unknown boson with mass 125.3 ± 0.6 GeV/c2 and ATLAS of a boson with mass 126.5 GeV/c2. Using the combined analysis of two interaction types (known as 'channels'), both experiments reached a local significance of 5-sigma—or less than a 1 in one million chance of error. When additional channels were taken into account, the CMS significance was reduced to 4.9-sigma.

The two teams had been working 'blinded' from each other from around late 2011 or early 2012, meaning they did not discuss their results with each other, providing additional certainty that any common finding was genuine validation of a particle. This level of evidence, confirmed independently by two separate teams and experiments, meets the formal level of proof required to announce a confirmed discovery. Source: Wikipedia

 

[Dinwar]

jt512, tell me how they can run multiple tests of a singleton fossil, or how many geologic maps are necessary before they become considered acceptable, and then we'll talk. Until then, this is nothing but what I call Physics Envy: the desire to force science as such to conform with the standards of one field, usually physics (hence the name). Unfortunately for you and your ilk, real science simply isn't done that way. But hey, I'm willing to be proven wrong--answer those two questions in the first sentence, or either one of them really, and I'll consider your idea.

What would it take for you to consider the notion that you're wrong? Would anything? If nothing would be sufficient, can you REALLY claim to be rational about this?

 

[jt512]

jt512, tell me how they can run multiple tests of a singleton fossil, or how many geologic maps are necessary before they become considered acceptable, and then we'll talk. Until then, this is nothing but what I call Physics Envy: the desire to force science as such to conform with the standards of one field, usually physics (hence the name). Unfortunately for you and your ilk, real science simply isn't done that way. But hey, I'm willing to be proven wrong--answer those two questions in the first sentence, or either one of them really, and I'll consider your idea.

What would it take for you to consider the notion that you're wrong? Would anything? If nothing would be sufficient, can you REALLY claim to be rational about this?

What would it take for you to stop caricaturing my position?