Measuring Doubt: Examples and Thought Experiments

In cricket an umpire is supposed to give the benefit of doubt to the batsmen. There is a very close analogy to that, as some would say, in real life. Consider the decision a judge or jury has to make at a trial. In criminal trials the term used is beyond a reasonable doubt and in civil cases it is more a case of preponderance of doubt (balance of probabilities). The consequences are more somber but what are these except different standards of benefits of doubt. Reasonably, in criminal cases we look for guilt beyond a reasonable doubt. The idea here is to let a criminal go rather than convict an innocent man. However, This lowers the overall accuracy of results. People have done crude experiments
http://en.wikipedia.org/wiki/Burden_of_proof

to determine what is beyond a reasonable doubt (70 to 75% surety). Better experiments can be done. But the more safe we play in one direction the more errors we make in the other. Accuracy over many cases and “beyond reasonable” doubt are two opposing forces.

In cricket the umpire is supposed to give the benefit of doubt to the batsmen. Yet he is valued for his accuracy. To get the best accuracy he should go with the balance of probabilities. If he thinks that the probability of being out is better than 50% he should give it out. However, this lowers the amount of the “benefit of doubt” he is willing to give the batsmen.

Consider the following thought experiment.
You have one of those old, big camera bodies with a shutter. There is a fair coin inside. We shake the box then go click. The shutter opens and closes. While the shutter is open we try to see whether the coin is head or tails. We record the result.

As long as the shutter is really slow we have very high confidence that we know whether the coin is head or tails. If we keep on increasing the shutter speed it will get harder for us to know with a high degree of confidence whether we saw a head or tail until the shutter is so fast that we can’t really judge and a random guess is the best we can do.

Now we are going to repeat the experiment let us say a 100 times. Each time the shutter speed is the same and very, very slow. We will get all the head and tails correctly. Our accuracy would be 100%. We would have no doubt about any of our calls.

Now we repeat the above experiment but make the shutter speed faster so that on any given shutter closing we are only 70% sure that we know whether it is a head or a tail. If we go with the best of our ability we will get it right close to 70 times.

If the shutter is very fast we might as well guess and get it right about 50 times.

In each case what was our level of doubt. In the first case we have no doubt at all, in the last case our doubt is total.

Now consider that we are told that if you are doubtful you should choose head. In both cases two and three we have some doubt and will choose head getting only. In both cases we will get it right about 50% of the time. In case two this has resulted in a loss of accuracy. Our accuracy in predicting has gone down by twenty percent because we are doubt intolerant.

Now consider the same experiment but each time the shutter speed is variable. So some are easy calls (that was surely a head etc.). about some you have a little bit of doubt and about some there is major doubt. If anytime you have even a little doubt you call a head. The result will be skewed towards heads. Your accuracy will suffer. If you want to maximize your accuracy you would have to let in an appropriate amount of doubt.


This is no different from a medical test where the test has some accuracy (not 100%) resulting in false positives and false negatives. You can usually change the threshold to make the test more biased towards false positives or false negatives but your accuracy also suffers.

So if the incentive is to be as accurate as possible then let in some doubt.
If you have not figured it out by now. Heads and tails corresponds to being out or not out. The fast shutter speed to a difficult decision the low shutter speed to an easy decision.

This is also no different from medical tests where we have false positives and false negatives (The test being the umpire). Usually the test can be made to tilt more towards false positives or false negatives if you are willing to give up some accuracy.