Back to Blog Landing

Seven Metrics That Matter When Measuring DevOps Success

Riverbed Technology December 11, 2017

Companies of all shapes and sizes are embracing DevOps to better compete in the digital era. Those with fairly advanced practices get higher-quality products to market much faster and continuously identify areas for improvement. Such practices ultimately lead to higher revenue, user productivity, and customer satisfaction.

DevOps Lifecycle

According to RightScale’s 2017 State of Cloud Report, 78% of surveyed companies reported some level of DevOps adoption. But why do some organizations remain on the sidelines? And for those who have gotten started, what’s preventing them for making better progress?

Often, it comes down to not having a proper measurement system. Without the right metrics, how can IT leaders prove the value of DevOps to the C-suite, let alone pave the way for continued investments to fine-tune their efforts?

This post is intended to provide a quick primer for evaluating DevOps success. You’ll also see how the right performance management platform provides the business-relevant insights needed for ongoing improvements.

Velocity: the most tangible benefit of DevOps success

Automation is the most visible element of the DevOps movement. So the linkage here in terms of velocity should be fairly obvious. And that is getting new application updates into users’ hands in the quickest, most reliable and secure manner possible. Doing so keeps products competitive with new features, and helps you maintain customer loyalty through enhanced engagement.

To that end, here are a couple of metrics for you to consider:

  1. Deployment frequency: This is how often you’re releasing new versions of applications, measured by the number of deployments over a given period of time. Puppet’s 2017 State of DevOps Report shows that high-performing DevOps practitioners deploy code updates 46x more frequently.
  2. Deployment lead time: How fast are your teams pushing updates from the point of code commit to an actual production environment? With traditional, monolithic applications, these change windows might last hours or even a couple of days, and were probably confined to non-peak hours. That’s no longer acceptable today. Again citing Puppet’s report, high performers experience change lead times of under an hour, or 440x faster than average.

Service quality: striking the balance between DevOps performance and speed

Speed is paramount with DevOps; there’s no questioning that. However, if new features or updates aren’t performing to users’ expectations, then that velocity is all for naught.

It’s equally important to ensure service quality—across releases and during times of change. Here are a couple of metrics that will inform you of whether you’re balancing speed and quality.

  1. Change failure rate: This is the percentage of releases that result in some type of failure. For example, ones that lead to unplanned outages, severe performance degradations, or require a hotfix, roll-back, etc. Those organizations with more mature DevOps practices not only ensure adequate pre-deployment test coverage, but also ensure app stability when pushed to production. According to a research study Riverbed conducted with ESG, three-fourths of organizations with mature performance management practices report that only 10% of app releases or fewer cause issues. By contrast, those with less advanced practices are only half as likely to report such a low frequency.
  2. Experts often have an MTTR under 30 minutes when addressing critical app issues.
    Experts solve performance issues much faster than less advanced organizations, which can prevent customer satisfaction issues and lost business.

    Number of errors or support tickets: How many bugs or performance issues are you encountering with a particular release? Are those numbers higher or lower than previous versions? It’s also important to track how your teams are learning about these incidents. Are they all user reported, or are your monitoring tools capturing them sooner?

  3. MTTR: When issues occur, it’s important to detect and resolve them as quickly as possible. That’s because every millisecond counts. Consider this: a second of sub-par performance can cost Amazon $1.6 billion annuallyESG’s research again shows us that those with mature performance management practices resolve issues faster: often in less than 30 minutes. That’s 4x faster than those with less advanced practices.

End-user experience: A critical success factor for DevOps

In addition to speed and quality, DevOps-oriented teams also need ways to measure the impact of changes. After all, you need to ensure your application updates are aligning with customer and business needs.

Two metrics that can help you do so are:

  1. Performance graph translating performance issues into financial impact
    Being able to translate the performance of individual application components into actual financial impact is key for DevOps success.

    Feature usage: What percentage of the targeted user base is leveraging a new feature or capability? Of that subset of users, what do their usage profiles look like? Looking at the number of sessions, completed transactions, and duration of each session will provide a starting point.

  2. Business impact: Usage is one thing, but how is that usage (or lack thereof) contributing to the business bottom-line? You should have tools in place that not only monitor the application experience for users, but also translate those usage metrics into actionable intelligence regarding impacts on productivity, revenue, and customer satisfaction.

Collecting metrics with the right DevOps monitoring technology

Of course, the above list is by no means exhaustive. But hopefully it provides a good starter list, and also prompts you to think of other metrics that matter.

Now, the other logical thing to address is how to capture these metrics. Many companies are utilizing application performance management (APM) solutions to either drive improvements to these metrics or simplify the process of collecting and reporting on them.

Long known for their value in production troubleshooting, many companies are now leveraging APM tools to marry the increased velocity and automation of DevOps with a greater level of quality assurance throughout the entire application lifecycle.

You can explore how APM can contribute to the success of DevOps practices by reading this blog post. But in short, you can:

  • Accelerate development: Detect and resolve bugs earlier in development, and bring more reliability to test scenarios by simulating application performance in various real-world environments.
  • Resolve issues before business is impacted: Look for tools that provide insights spanning user devices, apps, networks, and infrastructure. Such a holistic picture can decrease downtime by more than 80%.
  • Measure the impact of releases for more informed planning: Gain clear insights into the end-user experience to understand what features are popular, how they’re performing, and how that experience is impacting the business from a financial perspective.

    Measuring business impact of poor app user experience
    The right end-user performance monitoring tool can translate application consumption into business-relevant insights like impacts on user satisfaction, productivity, and revenue.

Riverbed solutions for DevOps

Riverbed is here to help you with your DevOps monitoring strategy. Our solutions are designed to provide you deep and continuous lifecycle insights into how your applications are performing, so you can consistently deliver the superior digital experiences today’s customers expect. Learn more by visiting our DevOps solution page.

You may also like

Three Ways Performance Monitoring Gives DevOps a Boost

In today's digital economy, businesses need to rapidly develop new apps, products, or services and collect timely feedback from customers.

Read More