Allan Kelly

About Allan Kelly

Allan Kelly has held just about every job in IT, these days he provides training and consulting in development management, processes & products, especially around Agile. He specializes in working with software product companies, aligning company strategy with products and processes. More about Allan at http://www.softwarestrategy.co.uk/allankelly

Testing triangles, pyramids and circles, and UAT

A few months ago Markus Gartner introduced me to the Testing Triangle, or Testing Pyramid. It looks like this:

If you Google you will find a few slightly different version and some go by the name of Testing Pyramid.

Now a question:

where did this come from? Who should I credit with the original? Markus thinks it mike be Mike Cohn but he’s not sure.

This triangle is actually pretty similar to a diagram I’ve been drawing for a while when I do Agile training:

But it occurs to me the triangle should be pushed to the side, and when you do that you can add some axis which add more information:

At the base the, Unit Tests, there are lots and lots of tests and they typically execute in milliseconds. I typically hear people say there are between two and four times as much test code (for unit tests) as production code.

As you rise up there are fewer tests, tests take longer to execute and therefore tend to be run less often. Also as you rise up it becomes more difficult to automate the tests, you can automate but it requires more effort and more co-ordination. Therefore manual testing continues to exist.

True, manual testing can never be eliminated entirely for various reasons (e.g. exploratory testing) but you can get very high levels of automated acceptance or system tests.

Cost is proportional to time – and manual testing is an order of magnitude more expensive than automated tests over anything other than the very short run – and therefore as tests take longer to run costs go up.

Now a word on UAT or Beta testing.

As far as I am concerned User Acceptance Testing is the same as proper Beta Testing. Both mean: showing a potentially finished product to real life users and getting their response.

The difference is: UAT tends to happen in corporate environments where users work for the company and will need to use the software. Beta testing tends to happen in software vendor environments and it means showing the software, even giving it to, potential users outside the company.

Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.

This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing.

And having users play with software means they are not doing their actual job so UAT is very expensive: expensive because it is manual and expensive because something else is not being done. Given this cost it is sensible to minimise UAT whenever possible.

In my experience most UAT phases (although not beta phases) are in effect an additional round of System Test and are frequently performed by Professional Testers. The fact that these professional testers are doing UAT is the give away. Professional Testers are not Users, they are Professional Testers.

(The other give away is that Professional Testers doing UAT are usually paid for by some group other than the IT department, i.e. IT test is not trusted, perhaps for good reason.)

More than once I have seen System Test / Acceptance Test cycles which are either completely absent or very poorly done. This necessities the need for a second round. Which is called UAT (possibly to hide the actual problem?)

I also see situations were Unit Test is poorly done or completely omitted.

If the low levels of this triangle are done well – and quality built in – then UAT should be reduced to a) real users, b) as near as possible a formality.

UAT is a very expensive way to find bugs, it also shows that something was missing in the development process. Either in coding or before that in understanding the users. It also, perhaps more worryingly, shows a failing in the earlier test steps.
 

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

Leave a Reply


− 6 = one



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close