MythBusters results, question about bias

[T'ai Chi]

I've been interested in the TV show MythBusters for a while, generally impressed by what they do, and it makes for entertaining TV.

I wrote this (http://www.statisticool.com/mythbusters.htm) a while ago on one of their episodes, which thankfully led me to the interesting work done by Provine on scientifically studying yawning.

Anyway, just now I did a rough tally of the MythBusters' total outcomes (see attached graph).

It got me thinking, could someone look at their results at say 'Wow! See, all those silly people believing in those myths, when they are just a bunch of bunk!', yet another person look at their results and say 'Well, their name is the MythBusters, not the MythConfirmers, so what, did you really expect to see any other result other than the debunking of soft-targets?' ?

Of course, it is entertainment, so I don't really get up in arms about the debate applied to a fun TV show. But in the 'real world', say you have an organization that is pro X. Wouldn't you expect to see studies 'demonstrating' the pro X result more often than not? And same argument applies to an organization that holds that anti-X position.

How do you personally make a decision on X if there are an equal number of scientifically good studies, each showing the pro X and anti-X position?

 

[JEROME DA GNOME]

But in the 'real world', say you have an organization that is pro X. Wouldn't you expect to see studies 'demonstrating' the pro X result more often than not? And same argument applies to an organization that holds that anti-X position.

One of the most reasoned statements I have seen posted on this forum.

Well done!

 

[Wowbagger]

They should be called "MythTesters", but it just doesn't sound as good as "Mythbusters". But, they are really after testing myths, for their validity. They are not just out to bust them.

The Mythbusters' conclusions are based on what is self-evident in each test. They really don't make any judgement calls - it either works or it does not.
And, when their answer is not quite clear enough, they admit it is "plausible". They don't shoehorn it into "busted" or "confirmed". More testing might be necessary, in that case.

Other things to add:
* They don't just put in a little token effort, either. These folks try their darndest to get the myths to work. They only give up after exhaustive effort.

* They are sometimes suprised at their results, more times than most would like to admit. If a result does not agree with their personal prediction, they "publish" anyway. They don't sweep it under a rug.

* Anyone with the proper resources and training (if necessary) can try to duplicate their results. They are out there for the community to scrutinize. They do not conduct their efforts in secret labs.

* Sometimes they get things wrong. (Hey, no one is perfect). But, they own-up to it! They retest and revisit whenever necessary, and sometimes the outcomes could be different. They publish the new results in its place, and talk about what they did wrong before, and what they learned since.

* Etc.

 

[Wowbagger]

How do you personally make a decision on X if there are an equal number of scientifically good studies, each showing the pro X and anti-X position?

Can you give me an example of this, just out of curiosity?

 

[Pyrts]

Of course, it is entertainment, so I don't really get up in arms about the debate applied to a fun TV show. But in the 'real world', say you have an organization that is pro X. Wouldn't you expect to see studies 'demonstrating' the pro X result more often than not? And same argument applies to an organization that holds that anti-X position.

It depends on how intellectually honest they are, and what the consequences of X are.

If there's very poor evidence but they just BELIEVE something to be true, then they may well ignore the negatives. We see this in selective memory and other similar things. If they can accept that the negative MIGHT be true (biased but somewhat open to change) and they keep records, then they will change their mind.

I've done it. Had to do it (academically and scientifically) last week. Luckily I wasn't stupid enough to write a paper on the thing. That kind of forces your position.

How do you personally make a decision on X if there are an equal number of scientifically good studies, each showing the pro X and anti-X position?

Speaking as someone with experience in this, you generally don't unless there are specific conditions.

Let's say there's a controversial new treatment and half say it's a placebo effect and half say it works at a 75% effective rate. Then you weigh how important it is to you, personally (is the pain/whatever worth a 75% chance), how it will impact you (cost, time, trouble), and how close a match you are to the patients in the successful study (are they all young and you're over 50?)

Now... the Mythbusters' choices are actually fairly clear. They look for myths were people are claiming that basic laws of physics or physiology are possibly being violated.

The average human's understanding of physics and physiology is pretty poor (to judge from some of the classes I've taught.) So there tend to be more wild claims than observations of true but extraordinary events. In addition, each of these stories is often framed as a "cautionary tale", which means it may have been elaborated on.

So we have three conditions (bad observations, poor understanding of physics/physiology, tale elaboration) that exist which open avenues for "bad data" to creep into the "myth."

They do cherry pick for things they CAN do and CAN test, but I think it's a fair pick. I do think they go for myths that make them say "naawwwww... no... really????" rather than take the "well,that's a DUH" ones or "Captain Obvious Misses The Point" ones.

 

[Hawk one]

[snip]
Other things to add:
* They don't just put in a little token effort, either. These folks try their darndest to get the myths to work. They only give up after exhaustive effort.
[snip]

One good etc. to add:

* They figure out, demonstrate, and explain -why- a myth is either confirmed, plausible, or busted. Such as the quicksand experiment (will humans be fully sucked into it in mere minutes?), where it was revealed that the density of quicksand is big enough that humans will actually float like a buoyant.

 

[Wowbagger]

Another "etc." to add:

* They use measurement equipment they way they were designed to be used. Compare that to, say, The Ghost Hunters, for example, who seem to abhor instruction manuals and specification sheets.

 

[CapelDodger]

* They are sometimes suprised at their results, more times than most would like to admit. If a result does not agree with their personal prediction, they "publish" anyway. They don't sweep it under a rug.

If something's cool enough to become a myth, it's cool enough for TV, especially when it works.

Another to add :

* Imaginative practical minds. And the practical skills to go with them.

 

[rocketdodger]

I have been pretty disappointed in them on certain occaisions, however.

For example, in their katana myths episode, they completely either missed or ignored the documented fact that what gives a well made katana its cutting ability is a slicing action as opposed to a chopping one.

Even more disappointing was that the "expert" they got to demo for them didn't seem to know this either.

 

[Unalienable]

They make blunders all the time, but like you folks say, it's still entertaining television.

I think their single biggest fault is that they question a myth "It has been said that (X) has happened" and then they test this hypothesis by trying to replicate it, and then conclude that it's "busted" just because they failed to make it happen.

It's true that if they replicate it, they add some credibility to the story that (X) has happened. But if they cannot replicate it, they are still a far way from suggesting that (X) couldn't have happened.

They are clever, but they aren't engineering gods. Just because their Ming Dynasty Astronaut simulator couldn't get off the launchpad (after a day or two of fiddling) doesn't mean that generations of smart Chinese working on the problem couldn't figure it out.

 

[technoextreme]

They are clever, but they aren't engineering gods. Just because their Ming Dynasty Astronaut simulator couldn't get off the launchpad (after a day or two of fiddling) doesn't mean that generations of smart Chinese working on the problem couldn't figure it out.

No. It does. They had those rocket scientists. If they couldn't get the rocket off the ground it wasn't going to back in the Chinese time period. Not only that but Im fairly certain that their skill levels pretty much blow out anything generations of ancient Chinese scientists had. For crying out loud the myth involved sending a man into space using a chair. It doesn't take a rocket scientist to know that a chair isn't aerodynamic enough. In fact I would hazard a guess that I could bust 3/4 of the myths they perform using pen and paper.

For example, in their katana myths episode, they completely either missed or ignored the documented fact that what gives a well made katana its cutting ability is a slicing action as opposed to a chopping one.

Because it doesn't change anything.

Now... the Mythbusters' choices are actually fairly clear. They look for myths were people are claiming that basic laws of physics or physiology are possibly being violated.

Not really. They've done myths before where the truth was all ready known. Flipping a car (Which aparently they've retested) with jet wash was one such myth were they knew it could happen. Bug bomb bomb was another myth that they probably knew was true before they started testing.

 

[Unalienable]

No. It does. They had those rocket scientists. If they couldn't get the rocket off the ground it wasn't going to back in the Chinese time period.

You either have overconfidence in modern experts, or you underestimate ancient ingenuity.

There are many cases of ancient engineering which were complete mysteries for hundreds of years and we only been able to shed light on them in recent times. Modern engineers don't think like ancient engineers. They think in terms of cranes, chains, and oil--not in terms of slave-power, hemp, and banana peels. If you handicap modern experts so that they have only the materials available to the ancients, you better believe they are going to be highly inept.

By the way, of course I don't believe he went into space, but that's not necessary for the myth to get started. Truth be told, I don't actually even believe the myth at all. Even the idea that the guy could get high enough into the air so that people might believe that he didn't die seems quite farfetched. But I used that an example because even in that preposterous case they still didn't really nail the coffin shut.

 

[CFLarsen]

Anyway, just now I did a rough tally of the MythBusters' total outcomes (see attached graph).

Please list the outcomes. Which episodes yielded which results?

It got me thinking, could someone look at their results at say 'Wow! See, all those silly people believing in those myths, when they are just a bunch of bunk!', yet another person look at their results and say 'Well, their name is the MythBusters, not the MythConfirmers, so what, did you really expect to see any other result other than the debunking of soft-targets?' ?

Given your history of defending just about any woo belief, it is no wonder you don't mention the obvious:

That the majority of the myths people believe are just that: Myths.

They should be called "MythTesters", but it just doesn't sound as good as "Mythbusters". But, they are really after testing myths, for their validity. They are not just out to bust them.

On the contrary: Adam in particular often expresses a lot of excitement if a myth could be true. In the cases that involve explosions, both Adam and Jamie go out of their way to see if the myths are true.

* Sometimes they get things wrong. (Hey, no one is perfect). But, they own-up to it! They retest and revisit whenever necessary, and sometimes the outcomes could be different. They publish the new results in its place, and talk about what they did wrong before, and what they learned since.

I think they have even re-revisited a couple of myths, because new information came in.

Other things to add:
* They don't just put in a little token effort, either. These folks try their darndest to get the myths to work. They only give up after exhaustive effort.

They are clever, but they aren't engineering gods. Just because their Ming Dynasty Astronaut simulator couldn't get off the launchpad (after a day or two of fiddling) doesn't mean that generations of smart Chinese working on the problem couldn't figure it out.

Could be - but you must remember that the Mythbusters also lean on science of today. They don't just tinker with the stuff, the team (there are many more people involved than just Adam, Jamie, Kari, Grant and Tori) investigate the science behind it, and use it to build their rigs.

Generations of smart Chinese may have been able to figure it out, but the Mythbuster's test also drew on the science accumulated over centuries.

I'll put my money on science over generations of smart Chinese any day.

 

[MG1962]

Could be - but you must remember that the Mythbusters also lean on science of today. They don't just tinker with the stuff, the team (there are many more people involved than just Adam, Jamie, Kari, Grant and Tori) investigate the science behind it, and use it to build their rigs.

Generations of smart Chinese may have been able to figure it out, but the Mythbuster's test also drew on the science accumulated over centuries.

I'll put my money on science over generations of smart Chinese any day.

But sometimes they do get it very wrong - they have tried the ice bullet twice and failed. However it never seemed to have occured to them to try firing the bullet with an air rifle

And their frozen chicken through the windscreen was interesting - seeing they were trying to refute RAF research on the topic. They did revist this one and finally got it right second time around - so that was something.

And dont get me started on the soliders over the bridge myth - that was easily the biggest dropped ball I have ever seen them do

 

[CFLarsen]

But sometimes they do get it very wrong - they have tried the ice bullet twice and failed. However it never seemed to have occured to them to try firing the bullet with an air rifle

I haven't seen that one. Was the original claim that it was fired with an air rifle? They take great pains to recreate the myth as it is.

And their frozen chicken through the windscreen was interesting - seeing they were trying to refute RAF research on the topic. They did revist this one and finally got it right second time around - so that was something.

Indeed.

And dont get me started on the soliders over the bridge myth - that was easily the biggest dropped ball I have ever seen them do

What was wrong with that?

 

[Gravy]

Test results by season start here:
http://en.wikipedia.org/wiki/MythBusters_(season_1)

 

[MG1962]

I haven't seen that one. Was the original claim that it was fired with an air rifle? They take great pains to recreate the myth as it is.



Indeed.



What was wrong with that?

With the ice the bullet, the myth doesn't specify the type of weapon - And as they discovered a regular gun heats too much

With the troops on the bridge - they tested it on a suspension bridge. The first of which were seen in the West in the 19th century, though the only modern example of collapse was 1850 (Angers Bridge). It has been military doctrine since at least the middle ages. I think with a little effort, evidence could date back to the Roman era.

The mythbusters worked from the assumption of harmonics, when the real culprit for wooden bridges is stress from the weight distribution. A modern example of this was the Kansas City collapse - Although already dangerously overloaded due to an unintentional design flaw, the collapse was caused by the people on hanging bridge dancing in time with the music

 

[Big Les]

But sometimes they do get it very wrong - they have tried the ice bullet twice and failed. However it never seemed to have occured to them to try firing the bullet with an air rifle

Now that would be interesting. Air weapons seem to be pretty much off the American radar, but they're historically very popular here and several C19th militaries either trialled or (one of them) briefly adopted, gas or air-powered rifles. I'd love to seem them revisit with a custom air weapon.

Bear in mind though that they were focussing upon the myth as it appears in the movies which have never IIRC featured anything but a conventional firearm firing custom "ice-bullets". They then thought to vary the bullet material, but not the weapon, which I agree was a failure of imagination.

And their frozen chicken through the windscreen was interesting - seeing they were trying to refute RAF research on the topic. They did revist this one and finally got it right second time around - so that was something.

That's the thing, they are only a TV show, but they are far more responible and intellectually honest than any other show - they come back and try again. The constraints of TV are many, so I think they make a pretty good fist of it. At least they don't outright fake their "experiments" like Brainiac have done.

And dont get me started on the soliders over the bridge myth - that was easily the biggest dropped ball I have ever seen them do

Can't pass comment on that - too physicsy for my little brain. But the same applies - they try hard, they get some vaguely meaningful results at least some of the time. That's pretty good for TV.

As to the katana slicing v chopping thing, I agree that in the tests I remember them doing (machine-gun barrel, another sword etc), simulating a biomechanically-delivered cut by a professional wouldn't have made any difference. On the other hand based on what was shown they seem to have been a bit misled on the "expert" they chose. Do bear in mind that production exigencies often see expert advice sidelined, even if Adam and Jamie would want to defer to said expert.

 

[MG1962]

That's the thing, they are only a TV show, but they are far more responible and intellectually honest than any other show - they come back and try again. The constraints of TV are many, so I think they make a pretty good fist of it. At least they don't outright fake their "experiments" like Brainiac have done..

With the chicken story, I didn't see dishonesty - but a sense of lazyness. I saw a documentary a few years ago in which the RAF spent a fortune developing an anti-bird stike windscreen. When the test rig was ready to go, they hauled of to the local supermaket, bought frozen chickens out of the freezer - Fired them at the windscreens and destroyed every one of them. For the life of them they could not figure out how they had gotten the whole process so wrong - Finally they repeated the experiements with thawed chickens, and the new windscreens worked a complete treat

 

[volatile]

----