The missing bullet holes

Abraham Wald (31 October 1902 – 13 December 1950) was a Hungarian and American mathematician and statistician who contributed to decision theory, geometry and econometrics, and founded the field of sequential analysis.One of his well-known statistical works was written during World War II on how to minimize the damage to bomber aircraft and took into account the survivorship bias in his calculations. He spent his research career at Columbia University. He was the grandson of Rabbi Moshe Shmuel Glasner.

https://en.m.wikipedia.org/wiki/Abraham_Wald

The story of the missing bullet holes is a story of survivability and efficiency, showing us the logic behind thinking about cost as survival. The majority of this narrative is derived from Wald's writings. He describes his work, discusses the context and history of his research, and how it has been used to resolve medical decision-making and transportation safety issues. Armoring the planes too much is a problem; armoring them too little is also a problem. Somewhere in between, there's an optimum solution. There were more bullet holes in the fuselage, not so many in the engines. The armor said Wald shouldn't go where the bullet holes are. It needs to go where the bullets aren't: on the engines. Wald's insight was to ask: where are the missing holes? Wald was pretty sure he knew the missing bullet holes were on the missing planes. Wald's other advantage was his tendency toward abstraction. A mathematician is always asking, "What assumptions are you making?"The engine is a point of total vulnerability. Wald's recommendations were quickly put into effect and were still being used by the navy and the air force through the wars in Korea and Vietnam.

https://www.profound-deming.com/blog-1/the-story-of-abraham-wald

What is Survivorship bias ?

Survivorship bias or survival bias is the logical error of concentrating on entities that passed a selection process while overlooking those that did not. This can lead to incorrect conclusions because of incomplete data.

Survivorship bias is a form of sampling bias that can lead to overly optimistic beliefs because multiple failures are overlooked, such as when companies that no longer exist are excluded from analyses of financial performance. It can also lead to the false belief that the successes in a group have some special property, rather than just coincidence as in correlation "proves" causality.

https://en.m.wikipedia.org/wiki/Survivorship_bias

The myth

It makes a great story.

The problem of armoring planes is assigned to Wald. Along with the assignment, he is given a fair amount of statistical data regarding aircraft damage, for example the location of damage from hits by enemy aircraft. It happens that most of the damage is located on the fuselage and very little in the area around motors, and the military is expecting to add armor to the fuselage, where the density of hits is highest. "Not so fast," said Wald. "What you should really do is add armor around the motors! What you are forgetting is that the aircraft that are most damaged don't return. You don't see them. Hits by German shells are presumably distributed somewhat randomly. The number of damaged motors you are seeing is far less than randomness would produce, and that indicates that it is the motors that are the weak point." The advice is taken, and in fact Wald's techniques for interpreting aircraft damage statistics continue through two later conflicts.

The Internet loves this tale.

The reason for this excitement is that the aircraft damage is an example of what is known as "survivorship bias." This is a technical term for what we all know well: the dead don't often get to tell their side of the story, and yet sometimes it would be better if they did. The loss is the source of all kinds of misinformation, as the Internet will tell you emphatically. Including deceptive practices in selling hot stocks, which may explain much of the buzz.

Well, it's gratifying to see a great mathematician become a legend for good reasons, rather than bad. "MATHEMATICAL GENIUS SCORES AGAINST ARMY BRASS!" reads pretty well. After all, publicity about mathematicians typically concentrates on features most of us would rather not think about. But it would be much more gratifying if there were more truth to the story, or at least more reason for believing it. Some of us prefer our history lessons to be taken from the non-fiction shelves.

The story might well be true, and there is certainly, as we shall see, a solid germ of truth in it, but there is very little evidence for the best bits. The capsule biography of Wald is accurate, and although he might not have been the smartest man in the room, he was probably nearly always the most accomplished mathematician in the room, which counted for a lot. But ... most of the rest of the story is--to use a charitable phrase--"plausible reconstruction." There is extremely little source material for what Wald had to say about aircraft damage.

The true story, or at least part of it

The memoranda by Wald are severely technical. Not much drama at all. In particular, Wald says nothing about what the military should do to improve things. If I understand Wallis correctly, it was the general policy of the SRG^* to answer just the questions asked and never--well, hardly ever--attempt to offer advice on applications of what they discovered. Military decisions were made by the military.

^* The Statistical Research Group (SRG) was a research group at Columbia University focused on military problems during World War II. Abraham Wald, Allen Wallis, Herbert Solomon, Frederick Mosteller, George Stigler, Leonard Jimmie Savage and Milton Friedman were all part of the group in which 18 researchers participated. https://en.m.wikipedia.org/wiki/Statistical_Research_Group

There are eight items among the memoranda. Five of them deal with a single problem, estimating probabilities of an airplane's survival, given that it has already been hit. Its outstanding feature is that it offers a way to estimate damage on the planes that never returned. A kind of magic, indeed. One--just one--deals with the problem of vulnerability of different sections of an airplane, and this shares with the previous sections some impressive estimates. That is, as the Internet fiction suggests, both have to deal with the problem that downed planes aren't around to give evidence.

Please read the full article

https://www.ams.org/publicoutreach/feature-column/fc-2016-06