Survivorship Bias and Product Development

A couple of months ago I came across an interesting article by the author of ‘You Are Not So Smart‘ about a fallacy known as ‘Survivorship Bias‘ which Wikipedia defines as:

The logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that didn’t because of their lack of visibility.

I particularly liked the story describing how Abraham Wald helped the US military overcome an instance of this error when trying to work out where to place armour on their bomber planes:

The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all.

The mistake, which Wald saw instantly, was that the holes showed where the planes were strongest. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.

I find the concept interesting on its own but as I’m working at a product company I was trying to understand how applicable it is for my particular context.

With respect to a product I think you’d say that non survivors are people who started using your product and then gave at some stage for whatever reason. Those people may not reach out for help so you lose the opportunity to get their feedback about the problems with your product.

At the other end of the scale you have people who are very happy with your product (the survivors) and they will often give you feedback about things that they’d like to see in the product for which they may currently have found a work around for.

What interests me at the moment is understanding whether the survivors’ opinions about how to improve the product are the same as that of the non survivors – survivorship bias suggests this probably isn’t the case so we need to find a way to learn what’s hurting the non survivors.

One way to do this is to conduct user testing where we get potential users of our product to use it and then observe the pain points that they encounter. Although we don’t get to see in which areas people would be prepared to persevere to achieve their goal, it is still useful for identifying weak points.

Another thing that my colleagues and I have been doing is discussing times at which we felt like giving up while using the product but persevered and ended up being survivors.

The reason looking at this is interesting is that we tend to be affected by the affect heuristic and although we might be prepared to overlook a weakness of the product others might go elsewhere.
 

Reference: Survivorship Bias and Product Development from our JCG partner Mark Needham at the Mark Needham Blog blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

Leave a Reply


4 − three =



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close