Home » Software Development » The Test Waste Ratio

About Gil Zilberfeld

The Test Waste Ratio

Remember the days when bugs were just part of life? I mean a big part.

If you recall those dark ages, we’ve spent most of our time fixing bugs and introducing others. In our spare time, we got to work on new stuff.

Today is different, there are a lot less bugs. So we get to work more on the new stuff, right?

Or have we found a new replacement for them?
 
 

Test Maintenance Is the New Bug

You wake up one morning, and you discover that you spend 25% on maintaining tests. True story.

This doesn’t happen in one day, you feel more work on tests creeping up, and you feel it’s just natural. Part of the work.

Then one day you actually look at the numbers. And someone needs to pick you off the floor.

Nobody Writes Bad Tests Intentionally

So, where do these tests come from?

These tests are the creaky windows in the attic. One gust of wind, and you hear a loud bang. You need to go up there and fix them. Again.

Much like those windows, they don’t give you much protection. Since they break once in a while, you get accustomed to them creaking and banging, you don’t notice them anymore. Which is bad, since sometimes they let some real menaces into the house. Like real bugs. But you won’t know because you’re sure it’s the ol’ window moving in its place.

So until you face reality, it’s not clear how much they cost us. It could amount to a lot.

Back to Test World

Let’s call these bad tests the “wasteful” tests.

Which begs the question: If those cause waste, when are tests valuable? What exactly is this value?

This is hard to quantify. A test is valuable if it catches a bug. It gets points for that. So, ridiculous as it may sound, the time spent on fixing bugs the tests found is an indicator to the value of the test. We can call those tests the “valuable” tests.

Now, here’s the Test Waste ratio:
clip_image0027_thumb2

The denominator is the total time spent due to tests breaking, the valuable and wasteful. The numerator is the wasteful work.

If your wasted time is large, the ratio gets closer to 1. If it’s small, it gets closer to zero.

You want to get a lower ratio as possible. It’s not really up to you, but it’s an indicator. And those numbers can actually be measured.

So Much For Theory

What do you think? Does it make sense? Is it helpful in any way? Do I stand a chance for a Nobel prize?

Let me know.

PS. Next week, I’m going to be at Belgium Testing Days. Apart from the Spaceship I’m smuggling through customs, I’m going to be part of an interactive session, talking about the ROI of test automation.
 

Reference: The Test Waste Ratio from our JCG partner Gil Zilberfeld at the Geek Out of Water blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

 

One comment

  1. I would argue that tests can be useful in other, more subtle, ways then catching bugs.

    * Documentation of behavior, which should save time for the next developer to understand the system

    * A good test coverage gives the Courage to refactor code to make it Clean. Clean code should be easier to understand and improve further.

    * Tests should drive the design to a more testable state. A testable system is easier to run in isolation. It should lead to a system that is easier to investigate.

    This is all secondary effects to having the system under test, but they all contribute value. But sure, not all tests are valuable. it is still code and can be implemented in crappy ways. There is no silver bullet.

Leave a Reply

Your email address will not be published. Required fields are marked *

*


Want to take your Java Skills to the next level?
Grab our programming books for FREE!
  • Save time by leveraging our field-tested solutions to common problems.
  • The books cover a wide range of topics, from JPA and JUnit, to JMeter and Android.
  • Each book comes as a standalone guide (with source code provided), so that you use it as reference.
Last Step ...

Where should we send the free eBooks?

Good Work!
To download the books, please verify your email address by following the instructions found on the email we just sent you.