Back to Blog Landing

4 Steps: Hold Your SaaS Vendor Accountable to Application Performance SLAs

Craig Mackland-Rivera January 21, 2015


Many of my enterprise customers have turned to a SaaS model for apps such as office productivity, HR, and CRM. Rather than acquiring, implementing, and maintaining software and the associated infrastructure, they use a SaaS solution on a pay-as-you-go basis for the seats they need. The benefits of SaaS to IT are clear:

  • Faster time to deploy
  • Reduced operational and capital expenses
  • Smooth software and hardware upgrades
  • The ability to shift staff to more strategic IT projects

However, these IT benefits are outweighed if SaaS delivered applications don’t deliver a quality end user experience to the workforce relying on them. Although SaaS vendors offer Service Level Agreements (SLAs) governing availability goals, incident response time commitments, and penalties, these metrics are insufficient to guarantee excellent end user experience. Penalties paid at the end of the month don’t help the end user whose app is slow when they schedule a meeting, file an expense report, or look up a customer.

Professional services firm implements SaaS expense reporting and gets an earful

I recently helped one of my customers address these SLA short-comings during their implementation of Infor Expense Management, a SaaS-based timesheet and expense management application. The company, a professional services firm with over 10,000 staff located in 100 offices around the world, works with the largest enterprises in the world on projects which can last from several months up to a year. Staff assigned to projects must account for their time in order to ensure the project remains within budget, so tracking timesheets and expenses is fundamental to the top-line revenue of the company as well as workforce reimbursement.

Aside from the Aternity administrator, my key point of contact within the firm’s IT shop was the product owner responsible for the Infor application. Shortly after rolling out Infor, she was barraged by user complaints of poor performance, but her colleagues could provide no details other than ‘it takes forever to enter my time sheets and expense reports.’ The product owner had no visibility into which areas of the application were having performance issues, what users were doing within the application when they experienced the slow response, or who was affected other than those who reported the issues.

Step 1: Identify the critical business activities

It sounds obvious, but monitoring the overall availability of an application is one thing, but monitoring the application’s performance is another. An app doesn’t have to be unavailable to cause a major impact to workforce productivity. What counts is the end user’s experience as they use the app to execute the business activities for which they are responsible. Since the firm had been a long time user of Aternity, the firm’s Aternity administrator reached out to me for help in addressing the problem.

So, our first step was to identify business critical activities. Together with the Infor product owner, we mapped the user’s interaction with the Infor app to key workflows, such as ‘login,’ ‘edit report,’ ‘save report,’ ‘add attendees,’ ‘create new time sheet,’ etc. This is done using ‘Signatures,’ part of the functionality within Aternity’s Business Activity Analytics. All in all, we created 24 different business activities that represented the top workflows for which the Infor app was used.

Step 2: Determine what constitutes acceptable performance for the activities

This step involves another basic concept. No application vendor or SaaS provider can tell a business what constitutes acceptable performance. Only the business can determine the appropriate response time metrics for business transactions. Acceptable performance must be based on what the end users expect, which is often driven by customer expectations. Business-driven thresholds for business activities are the foundation of a meaningful, application performance-based SLA.

In this case, the Infor product owner identified the thresholds for each activity based on reasonable expected response times. For example, the threshold for the ‘login’ activity was set to 6 seconds.

Step 3: Monitor SLA compliance across the enterprise

With thresholds set for the 24 key Infor business activities, the firm could now proactively monitor the end user experience of the workforce.

As the SLA dashboard above shows, the application was only around 30-40 percent compliant, meaning that over 60 percent of transactions failed to meet the established performance thresholds. Outside of the U.S., performance was even worse.

Step 4: Isolate and resolve the cause of poor performance

Problem resolution for SaaS-delivered applications is a challenge because the enterprise doesn’t have access to the infrastructure on which the application runs. In this case, though, my customer used Aternity to identify a direct, positive correlation between poor response time for certain business activities and the amount of incoming network traffic.

After some investigation, Infor determined that the application was not caching requests properly. As the dashboard below shows, the change they made to the application over the weekend of October 18 improved performance by almost 40%.

The key to application performance SLAs for SaaS – The end user’s perspective

Going beyond the basic availability, incident response time, and penalty policies that many SaaS vendors offer as part of their SLA requires monitoring the application from the end user’s perspective. Even SaaS vendors which share infrastructure performance metrics fail to capture end user experience.

The problem is that metrics monitored from the perspective of the SaaS vendor’s infrastructure don’t reflect the actual end user experience as cloud-hosted applications render on their devices. Even when cloud providers use JavaScript Injection to embed code into a web application in order to get timings of what is occurring within a user’s browser, they do not obtain any correlating metrics about what’s happening on the user’s device.

Without this complete picture, enterprises lack information critical to rapid problem resolution. As this customer story shows, using Aternity is one way to get the true understanding of end user experience on which application performance SLAs can be based.

To learn more about Aternity’s unique value and End User Experience Management advantages, read the Aternity whitepaper on the EUEM Landscape.


You may also like

gartner digital workplace summit, employee experience matters
Employee Experience Matters at Gartner Digital Workplace Summit

We’re super-excited to sponsor the 2020 Gartner Digital Workplace Summit

Read More
Is Managing your Document Management Software Costing Your Firm Time and Money?

Since working for Aternity over the last few years, I’ve

Read More
end user experience monitoring, EUEM, digital experience monitoring, DEM
3 to 1: The Payback of an Aternity Investment

Digital transformation continues to drive growth in the enterprise software

Read More