Home » Software Development » Survivorship Bias and Product Development

About Mark Needham

Survivorship Bias and Product Development

A couple of months ago I came across an interesting article by the author of ‘You Are Not So Smart‘ about a fallacy known as ‘Survivorship Bias‘ which Wikipedia defines as:

The logical error of concentrating on the people or things that “survived” some process and inadvertently overlooking those that didn’t because of their lack of visibility.

I particularly liked the story describing how Abraham Wald helped the US military overcome an instance of this error when trying to work out where to place armour on their bomber planes:

The military looked at the bombers that had returned from enemy territory. They recorded where those planes had taken the most damage. Over and over again, they saw the bullet holes tended to accumulate along the wings, around the tail gunner, and down the center of the body. Wings. Body. Tail gunner. Naturally, the commanders wanted to put the thicker protection where they could clearly see the most damage, where the holes clustered. But Wald said no, that would be precisely the wrong decision. Putting the armor there wouldn’t improve their chances at all.

The mistake, which Wald saw instantly, was that the holes showed where the planes were strongest. The holes showed where a bomber could be shot and still survive the flight home, Wald explained. After all, here they were, holes and all. It was the planes that weren’t there that needed extra protection, and they had needed it in places that these planes had not. The holes in the surviving planes actually revealed the locations that needed the least additional armor. Look at where the survivors are unharmed, he said, and that’s where these bombers are most vulnerable; that’s where the planes that didn’t make it back were hit.

I find the concept interesting on its own but as I’m working at a product company I was trying to understand how applicable it is for my particular context.

With respect to a product I think you’d say that non survivors are people who started using your product and then gave at some stage for whatever reason. Those people may not reach out for help so you lose the opportunity to get their feedback about the problems with your product.

At the other end of the scale you have people who are very happy with your product (the survivors) and they will often give you feedback about things that they’d like to see in the product for which they may currently have found a work around for.

What interests me at the moment is understanding whether the survivors’ opinions about how to improve the product are the same as that of the non survivors – survivorship bias suggests this probably isn’t the case so we need to find a way to learn what’s hurting the non survivors.

One way to do this is to conduct user testing where we get potential users of our product to use it and then observe the pain points that they encounter. Although we don’t get to see in which areas people would be prepared to persevere to achieve their goal, it is still useful for identifying weak points.

Another thing that my colleagues and I have been doing is discussing times at which we felt like giving up while using the product but persevered and ended up being survivors.

The reason looking at this is interesting is that we tend to be affected by the affect heuristic and although we might be prepared to overlook a weakness of the product others might go elsewhere.
 

Reference: Survivorship Bias and Product Development from our JCG partner Mark Needham at the Mark Needham Blog blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*


Want to take your Java Skills to the next level?
Grab our programming books for FREE!
  • Save time by leveraging our field-tested solutions to common problems.
  • The books cover a wide range of topics, from JPA and JUnit, to JMeter and Android.
  • Each book comes as a standalone guide (with source code provided), so that you use it as reference.
Last Step ...

Where should we send the free eBooks?

Good Work!
To download the books, please verify your email address by following the instructions found on the email we just sent you.