Allan Kelly

About Allan Kelly

Allan Kelly has held just about every job in IT, these days he provides training and consulting in development management, processes & products, especially around Agile. He specializes in working with software product companies, aligning company strategy with products and processes. More about Allan at http://www.softwarestrategy.co.uk/allankelly

Testing triangles, pyramids and circles, and UAT

A few months ago Markus Gartner introduced me to the Testing Triangle, or Testing Pyramid. It looks like this:

If you Google you will find a few slightly different version and some go by the name of Testing Pyramid.

Now a question:

where did this come from? Who should I credit with the original? Markus thinks it mike be Mike Cohn but he’s not sure.

This triangle is actually pretty similar to a diagram I’ve been drawing for a while when I do Agile training:

But it occurs to me the triangle should be pushed to the side, and when you do that you can add some axis which add more information:

At the base the, Unit Tests, there are lots and lots of tests and they typically execute in milliseconds. I typically hear people say there are between two and four times as much test code (for unit tests) as production code.

As you rise up there are fewer tests, tests take longer to execute and therefore tend to be run less often. Also as you rise up it becomes more difficult to automate the tests, you can automate but it requires more effort and more co-ordination. Therefore manual testing continues to exist.

True, manual testing can never be eliminated entirely for various reasons (e.g. exploratory testing) but you can get very high levels of automated acceptance or system tests.

Cost is proportional to time – and manual testing is an order of magnitude more expensive than automated tests over anything other than the very short run – and therefore as tests take longer to run costs go up.

Now a word on UAT or Beta testing.

As far as I am concerned User Acceptance Testing is the same as proper Beta Testing. Both mean: showing a potentially finished product to real life users and getting their response.

The difference is: UAT tends to happen in corporate environments where users work for the company and will need to use the software. Beta testing tends to happen in software vendor environments and it means showing the software, even giving it to, potential users outside the company.

Thus: UAT and Beta testing can only truly be performed by USERS (yes I am shouting). If you have professional testers performing it then it is in effect a form of System Testing.

This also means UAT/Beta cannot be automated because it is about getting real life users to user the software and get their feedback. If users delegate the task to a machine then it is some other form of testing.

And having users play with software means they are not doing their actual job so UAT is very expensive: expensive because it is manual and expensive because something else is not being done. Given this cost it is sensible to minimise UAT whenever possible.

In my experience most UAT phases (although not beta phases) are in effect an additional round of System Test and are frequently performed by Professional Testers. The fact that these professional testers are doing UAT is the give away. Professional Testers are not Users, they are Professional Testers.

(The other give away is that Professional Testers doing UAT are usually paid for by some group other than the IT department, i.e. IT test is not trusted, perhaps for good reason.)

More than once I have seen System Test / Acceptance Test cycles which are either completely absent or very poorly done. This necessities the need for a second round. Which is called UAT (possibly to hide the actual problem?)

I also see situations were Unit Test is poorly done or completely omitted.

If the low levels of this triangle are done well – and quality built in – then UAT should be reduced to a) real users, b) as near as possible a formality.

UAT is a very expensive way to find bugs, it also shows that something was missing in the development process. Either in coding or before that in understanding the users. It also, perhaps more worryingly, shows a failing in the earlier test steps.
 

Related Whitepaper:

Software Architecture

This guide will introduce you to the world of Software Architecture!

This 162 page guide will cover topics within the field of software architecture including: software architecture as a solution balancing the concerns of different stakeholders, quality assurance, methods to describe and evaluate architectures, the influence of architecture on reuse, and the life cycle of a system and its architecture. This guide concludes with a comparison between the professions of software architect and software engineer.

Get it Now!  

Leave a Reply


− six = 3



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books