Very hard question in online test: all answers are wrong?

[Ivor the Engineer]

I've just taken the Bang goes the Theory Big Risk Test. All answers were multiple choice. One of the questions was as follows:

A criminal hides in a room with 99 innocent people. You have a lie detector that correctly classifies 95% of people. You pick someone at random, wire them up to the machine, and ask them if they are a criminal. They say no, but the machine goes ‘ping’ and says the person is lying. What is the chance that you have caught the criminal?

I picked the answer nearest to the one I calculated from those available (95%, 83%, 17% and 5%, IIRC):

17%

The value I calculated was...

...the PPV = 0.01*0.95/(0.01*0.95+0.99*0.05) = 16.1%, or 16% to two sf.

Have I missed something or are all the available options wrong, albeit only by a small amount?

 

[RobDegraves]

It's 95%

The machine is 95% accurate.

The person said he's not a criminal and he is lying... therefore if the machine is accurate, he's is a criminal. It's a trick question.

 

[Beth]

I've just taken the Bang goes the Theory Big Risk Test. All answers were multiple choice. One of the questions was as follows:



I picked the answer nearest to the one I calculated from those available (95%, 83%, 17% and 5%, IIRC):

17%

The value I calculated was...

...the PPV = 0.01*0.95/(0.01*0.95+0.99*0.05) = 16.1%, or 16% to two sf.

Have I missed something or are all the available options wrong, albeit only by a small amount?

I don't think you've missed anything, you've computed the exact probability.

I think what they've done is round to integers. The test is expected to identify 6 people out of the 100 as criminals. One of them actually is the criminal. Thus, the probability is 1/6 ~ 17%. Your computation also takes into account the small chance that more or less than 6 people are identified as criminals.

 

[Perpetual Student]

I believe 0.01/(0.01*0.95+0.99*0.05) = .1695
is the correct answer.

 

[The_Animus]

I agree with Beth

 

[vwgub]

I think you are missing false positive and false negatives. I am trying to figure out the math but my best guest would be 83

 

[69dodge]

I get 19/118 = 16.1016949...%.

So, the same as Ivor the Engineer.

 

[69dodge]

The machine is 95% accurate.

That doesn't mean what you think it means.

It means, if the machine tests someone who is lying, with probability 95% it will say he's lying and with probability 5% it will say he's telling the truth. Likewise, if the machine tests someone who is telling the truth, with probability 95% it will say he's telling the truth and with probability 5% it will say he's lying.

Suppose you're telling the truth, and the guy running the machine knows it. Perhaps he asked you, "what color is the sky?" and you said, "blue". There is a 5% chance that the machine will misclassify you and say that you're lying. If that happens, the proper conclusion is that the machine must have messed up this time; the proper conclusion is not that you're probably lying about the sky being blue.

The prior probability of lying needs to be taken into account.

 

[ThunderChunky]

Me too. Baye's Law FTW.

 

[vwgub]

I think you are missing false positive and false negatives. I am trying to figure out the math but my best guest would be 83

Misread the original question I would also get 17 percent based off the question as asked.

 

[69dodge]

I believe 0.01/(0.01*0.95+0.99*0.05) = .1695
is the correct answer.

Can you explain how you got that?

 

[marplots]

I'd to it this way. Say you tested all 100 people. There would be 100 results. 5 would be incorrect and 95 would be correct (given in the problem). What is the probability that your criminal would be in the larger group? 95%

If he is in the larger pool, the test is giving the right answer, as it will for anyone in that group. So I'd go with 95%. I don't think the joint probability with the additional information should matter.

 

[Professor Yaffle]

For those thinking the answer is 95%... its a bit more complicated than that...

http://www.badscience.net/2006/12/crystal-balls-and-positive-predictive-values/

http://www.badscience.net/2009/02/datamining-would-be-lovely-if-it-worked/

 

[Dilb]

Can you explain how you got that?

I'm pretty sure that's just a small error. Perpetual Student is doing what Ivor et. al. are doing, except incorrectly using 0.01, rather than 0.01*95%. It's ignoring the possibility of the detector being wrong about the liar, but only in the numerator.

 

[RobDegraves]

OK... about the 95%, I think it has a bit to do with how you establish odds and how you read the question.

Let's look at flipping coins for a clearer example if you don't mind.

If I flip a coin once, the odds of getting heads is 50%.

If I flip a coin a hundred times, I should get heads about 50 times.

The odds of getting heads on throw 59 though... are much lower. Or, the odds of getting 99 heads is lower, etc.

However, each specific throw is still 50%, no matter what happened before.

So... look at your problem and how it is worded, specifically this part.

You pick someone at random, wire them up to the machine, and ask them if they are a criminal. They say no, but the machine goes ‘ping’ and says the person is lying. What is the chance that you have caught the criminal?

In this specific instance... you have a 95% chance of having caught the criminal.


I hope that explains my reasoning.

 

[69dodge]

Say you tested all 100 people. There would be 100 results. 5 would be incorrect and 95 would be correct (given in the problem).

Yes.

What is the probability that your criminal would be in the larger group? 95%

Yes.

If he is in the larger pool, the test is giving the right answer, as it will for anyone in that group.

No.

Let's say the criminal is definitely in the larger group, and so he fails the test. That doesn't mean that this person who failed the test is the criminal. Five other people besides the criminal also failed. This person is probably one of them, simply because there are more of them.

 

[69dodge]

Let's look at flipping coins for a clearer example if you don't mind.

If I flip a coin once, the odds of getting heads is 50%.

If I flip a coin a hundred times, I should get heads about 50 times.

Yes.

The odds of getting heads on throw 59 though... are much lower.

No, they're 50% too.

Or, the odds of getting 99 heads is lower, etc.

Yes.

However, each specific throw is still 50%, no matter what happened before.

Yes.

So... look at your problem and how it is worded, specifically this part.

You pick someone at random, wire them up to the machine, and ask them if they are a criminal. They say no, but the machine goes ‘ping’ and says the person is lying. What is the chance that you have caught the criminal?

In this specific instance... you have a 95% chance of having caught the criminal.

I hope that explains my reasoning.

I don't see the connection between this problem and your examples of the coin.

 

[Illustronic]

I have a 95% chance of getting caught lying to my wife, but there is only a 1% chance of that.

 

[yomero]

I'd to it this way. Say you tested all 100 people. There would be 100 results. 5 would be incorrect and 95 would be correct (given in the problem). What is the probability that your criminal would be in the larger group? 95%

If he is in the larger pool, the test is giving the right answer, as it will for anyone in that group. So I'd go with 95%. I don't think the joint probability with the additional information should matter.

I know very little about statistics, but I get that there is only a 16.66% chance approx. that a person failing the test is actually a criminal. If all 100 suspects are tested, approx. 94 would pass the test. We can expect the machine to go ''ping'' 6 times, once for the criminal and 5 times for innocent persons. So, if only one of those 6 selected by the machine is actually guilty, that would be 1 out of 6, or 16.66%

I have not taken into account the possibility (5 out of 100) that the criminal could pass the test. I don't know the math to do that.

 

[psionl0]

Presumably all 100 people involved would deny being the criminal. So the chance that the machine correctly identified the criminal would depend on how reliable the machine is - 95%.