Survivorship Bias and Product Development

A couple of months ago I came across an interesting article by the author of ‘You Are Not So Smart‘ about a fallacy known as ‘Survivorship Bias‘ which Wikipedia defines as:

The logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that didn’t because of their lack of visibility.

I particularly liked the story describing how Abraham Wald helped the US military overcome an instance of this error when trying to work out where to place armour on their bomber planes:

The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all.

The mistake, which Wald saw instantly, was that the holes showed where the planes were strongest. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.

I find the concept interesting on its own but as I’m working at a product company I was trying to understand how applicable it is for my particular context.

With respect to a product I think you’d say that non survivors are people who started using your product and then gave at some stage for whatever reason. Those people may not reach out for help so you lose the opportunity to get their feedback about the problems with your product.

At the other end of the scale you have people who are very happy with your product (the survivors) and they will often give you feedback about things that they’d like to see in the product for which they may currently have found a work around for.

What interests me at the moment is understanding whether the survivors’ opinions about how to improve the product are the same as that of the non survivors – survivorship bias suggests this probably isn’t the case so we need to find a way to learn what’s hurting the non survivors.

One way to do this is to conduct user testing where we get potential users of our product to use it and then observe the pain points that they encounter. Although we don’t get to see in which areas people would be prepared to persevere to achieve their goal, it is still useful for identifying weak points.

Another thing that my colleagues and I have been doing is discussing times at which we felt like giving up while using the product but persevered and ended up being survivors.

The reason looking at this is interesting is that we tend to be affected by the affect heuristic and although we might be prepared to overlook a weakness of the product others might go elsewhere.

Reference: Survivorship Bias and Product Development from our JCG partner Mark Needham at the Mark Needham Blog blog.
Related Whitepaper:

Software Architecture

This guide will introduce you to the world of Software Architecture!

This 162 page guide will cover topics within the field of software architecture including: software architecture as a solution balancing the concerns of different stakeholders, quality assurance, methods to describe and evaluate architectures, the influence of architecture on reuse, and the life cycle of a system and its architecture. This guide concludes with a comparison between the professions of software architect and software engineer.

Get it Now!  

Leave a Reply

seven + = 11

Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books