Measuring the Cloud: Metrics for the Next Generation

More and more organizations are turning to cloud services to reduce operational cost, transfer risk to service providers, and increase workforce mobility. Such change raises questions that must be answered. How do today’s leaders decide which services to migrate to the cloud and which to put in the parking lot for the next pass? How do they determine which vendors to use and how to measure their performance? Does moving the service to the cloud improve the customer experience?

Measure the Customer Experience

While there are some measures that can be modeled across industries, there are those that necessitate focusing on individual outcomes related to the customer’s business model. Have you ever heard the comment “But, we’re different!”? The customer’s desired outcomes (which you can identify through surveys and interviews) are where most of the “we’re different” comes from. Customers want to know how a given decision will affect them. So, when making any decision regarding technology, remember to answer the “So what?” question. Say you’ve decided to move to the cloud. So what? What will this do for your customers, and once you’ve migrated to the cloud, does the solution still meet your customers’ expectations?

To answer the “So what?” questions, develop service metrics that relate to your customers’ and stakeholder’s critical success factors. But recognize that these may change over time and the ability to adapt will make all the difference. Revisit these metrics periodically to adjust for changes in the economy, technology, and industry. Also, establish metrics for the services as well as the processes and technology that support those services. Each metric, measurement, and key performance indicator (KPIs) should relate back to the customer’s desired end result. At the end of the day, customers and stakeholders care more about what the service is doing for them. Getting caught up in metrics that do not relate back to the customer experience is a waste of time and resources.

Technology Test Drive

Before you buy that new sofa, wouldn’t you like to sit on it? The same mentality applies to the cloud. Do your homework before you bring in vendors. Research vendors online, talk to vendors, ask for demos, and consult your current customers. Once you’ve gotten down to your top two or three vendors, remember:

• Use core concepts of service management; 
• Demand technology providers provide metrics transparency; 
• Understand the security and privacy implications of moving the service to the cloud; and 
• Keep in mind that even if you do not provide the service, you are accountable for it.

Importance of Measurement

Should you have measurements and metrics for your cloud services? Of course. You can’t manage what you can’t control, you can’t control what you can’t measure, and you can’t measure what you can’t define. Without metrics, we would lose control of our services and we would have no way of knowing where to improve. When migrating to the cloud, there are risks involved, namely security, privacy, and the safety of intellectual property. So, it’s pretty easy to see why we should measure cloud computing.

The when, what, where, and how questions are not that easy to answer, and these answers will vary from organization to organization. However, it is vital that we hold our cloud services vendors accountable through metrics, SLAs, and other contractual mechanisms. They enable us to have better control over the services we provide to our customers and the support we provide for those services. After all, as we’ve already noted, we are still accountable for cloud services, even though we aren’t the service provider.

Cloud metrics also enable IT organizations to forecast future performance. By tracking data trends, we can see whether performance is degrading, improving, or flat. There are several different ways to visualize the data, whether you use advanced business intelligence tools or simple spreadsheets to collect data from your service providers. Using the information they provide, you can identify quality improvement areas, track usage trends, pinpoint areas for cost reduction, and manage risks.

What Measures Matter?

The right answer to this question is, “It depends.” However, the high-level metrics that matter to almost all organizations include customer, cost, and effectiveness. (Quality is also extremely important, but it falls under both customer and effectiveness.) We talked earlier about the value of speaking with your customers and determining what is important to them. This dialogue should continue throughout the service lifecycle by way of satisfaction surveys. These surveys don’t have to be elaborate, and there are free surveying tools out there on the web to facilitate development; alternatively, if your organization has a portal, many vendors have survey features and modules you can leverage.

The table below provides a list of sample metrics by category, some of which are probably already in your SLAs and vendor contracts. Keep in mind that this is not a complete list and it will vary by organization. Be sure to limit the list to those metrics that directly relate to the business outcomes you seek. Don’t just measure for the sake of measuring.

Recap: How to Measure the Cloud

1. Talk to your customers. Make sure you understand their values and their visions. 
2. Perform a service assessment to generate real data on what matters most to your customers and organization. Answer the “So what?” questions. 
3. Identify the critical success factors and key performance indicators to measure during the design phase.
4. Relate those CSFs and KPIs to cost. 
5. Set up your service acceptance criteria, which should specify thresholds that you require your vendors to meet. This will help you select a vendor. 
6. Evaluate the various cloud solutions (private, public, or hybrid) and then determine which vendors that can provide that service. 
7. Add the service provider you select—including the vendor’s information, SLAs, and other key metrics—to the service portfolio. 
8. Measure after implementation to get a baseline measurement of the new service. 
9. Use the data, information, and knowledge to hold your cloud computing vendor accountable and improve any components of the service. 
10. Periodically review metrics, CSFs, and KPIs. Adjust as needed.

As the Millennial generation enters the workforce, they expect the cloud services that enable their smartphones and tablets to function as work tools. To meet these expectations, you need to be able to ensure that your cloud services are meeting their expectations. The ability to adapt and overcome obstacles has long been the hallmark of successful companies, and it will continue to be so as organizations move to the cloud.

Jason Rosenfeld is the ITSM practice lead at Cask, LLC. He has fifteen years of hand-on ITSM experience with clients in the defense, transportation, finance, and pharmaceutical industries. Jason is the co-membership chair of the itSMF USA’s San Diego LIG and is currently working with Carnegie Mellon University and other industry leaders on the Cloud Service Measurement Index Consortium (CSMIC) to build a common measurement framework for cloud computing across industries.