Jim Bird

About Jim Bird

Jim is an experienced CTO, software development manager and project manager, who has worked on high-performance, high-reliability mission-critical systems for many years, as well as building software development tools. His current interests include scaling Lean and Agile software development methodologies, software security and software assurance.

How do you measure Devops?

If you’re trying to convince yourself (or the team or management) that your operations program needs to be changed for the better, and that trying a Devops approach makes sense – or that your operations organization is improving, and that whatever changes you have made actually make a difference – you have to measure something(s). But what?

Measuring Culture

John Clapham at Nokia suggests that you should try to measure how healthy your operations culture is. At the Devops Days conference this year in London he talked about ways to measure and monitor culture – behaviour, attitudes and values – to
 
determine whether people were focused on the “right things”, and to assess the team’s motivation and satisfaction. Nokia had already started a Devops program, and wanted to see whether the momentum for change and improvement was still there after the initial push and evangelism had worn off. So they came up with a set of vital signs that they felt would capture the important behaviours and attitudes:

  1. Cycle time – time from development to deployment in production. Are we moving faster, or fast enough?
  2. Shared purpose – do people all share/believe in the same goals, believe in improving how development and ops work together?
  3. Motivation – does everyone care about what they are doing?
  4. Collaboration – are people working together willingly?
  5. Effectiveness – is everyone’s time being spent in a useful way? How much time is being wasted?

Cycle time is the only measure that is relatively easy to measure and report. The rest are highly subjective and fuzzy. Nokia tried to collect this information through a questionnaire that asked questions like: Do you believe there are opportunities to improve ways of working? How much time do you spend on stability, overhead, improvements, innovation? What’s in your way: lack of time, pressure to focus on features, poor tools, lack of management support, nothing…?

Operations Vital Signs that you Can and Should Measure

Clapham’s closing question was: “What vital signs would you look for?”

I’m not convinced that you can measure an organization’s cultural effectiveness, or that it would be really useful if you could. You can’t tell from a wishy-washy questionnaire whether change is making a real difference to the organization’s effectiveness and you are on the right track; or help you understand what you need to change and what the impact of change would be on the bottom line (or the top line). To do this you need concrete and results-based measurements which point out strong points and weaknesses, and that you can use to make a case for change, or justify your decisions.

Puppet Labs and IT Revolution Press have recently published a “State of Devops Report”, which is full of interesting data. The report stresses the importance of metrics in understanding how your organization is performing and why a Devops program is, or would be, worthwhile. They provide a list of objective measures, broken down into two major types.

Agility and reliability metrics:

  1. Deployment rate/frequency
  2. Change lead time – how long it takes to get a change approved and into production
  3. Change failure rate (John Allspaw’s brilliant presentation “Ops Meta-Metrics” explains the importance of correlating deployment frequency/size/type and failures – type and severity – in production)
  4. Mean time to recover (and mean time to detect)

Functional metrics:

  1. Test cycle time – how long does it take to test a change?
  2. Deployment time – how long does it take to roll out a new change once tested and approved?
  3. Defect rate in production (defect escape rate)
  4. Helpdesk ticket counts – how much time is spent firefighting?

There are two other important measures that are missing from this list:

  1. Operations costs
  2. Employee retention – a key measure of whether people are happy

Measuring the success of a DevOps program is simple:
If you aren’t saving money…
If you can’t make change easier and faster…
If you don’t improve quality and reliability and your organization’s ability to respond to problems…
If you can’t keep good people…
… then whatever you’re doing is not working or you’re not doing it right. It doesn’t matter if you are “doing DevOps” or using certain tools or if people seem to be more collaborative or believe that they have a greater sense of shared purpose. What matters is the outcome. Make sure that you’re measuring the right things – so that you know that you are doing the right things.
 

Reference: How do you measure Devops? from our JCG partner Jim Bird at the Building Real Software blog.
Related Whitepaper:

Java Essential Training

Author David Gassner explores Java SE (Standard Edition), the language used to build mobile apps for Android devices, enterprise server applications, and more!

The course demonstrates how to install both Java and the Eclipse IDE and dives into the particulars of programming. The course also explains the fundamentals of Java, from creating simple variables, assigning values, and declaring methods to working with strings, arrays, and subclasses; reading and writing to text files; and implementing object oriented programming concepts. Exercise files are included with the course.

Get it Now!  

Leave a Reply


four − 2 =



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

15,153 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books