February 11, 2008

While it’s intuitive that end user experience is the most accurate measure of the quality and reliability of IT services, it’s often much less clear how to measure it. Moreover, there is no standard template for integrating an evaluation of user experience into the myriad of other objects monitored and measured. The process can be quite simple though in a scientific context. The goal is to build a picture of cause and effect in our environment that employs end user experience as the overall barometer of application performance and links it to all the potential service delivery problems. In statistical terms the end user experience is our Independent Variable. All the other things that can go wrong are our Dependent Variables, such as a system going down, running out of space, no response from web server or DB, network connectivity issues, etc. Ideally, we want to build a visualization of Independent and Dependent variables together, so that we immediately see the cause and effect relationship between end user experience and measures from many other application and network performance sources. These typically include hardware, operating system, and application performance measures, along with network monitoring and infrastructure tests like PING and port checks, etc. A really good SLA (Service Level Agreement) will include an accurate measure of the end user experience plus criteria that can impact service delivery, thus it’s a cause and effect picture. It not only tells us when we aren’t performing, the good SLA also suggests answers to the question “Why?” when service delivery suffers.

Capturing the elusive Independent Variable is our first goal. Measuring end user experience really means doing something a user does and evaluating success or failure and the time required. Using the example of a common web application model, in Longitude we would simulate a transaction that logs into the web site, navigates to some page and performs some transaction. The result of our Internet solution test becomes our Independent Variable. We can also measure components of end user experience separately by performing additional transaction monitoring tests that measure the web server response and the database response to a query. These are the first components added to our hypothetical SLA. Our list of Dependent Variables includes the potential impediments to service delivery, such as System resources exhausted, network bandwidth consumed, transaction rates, and more. The goal here is to capture enough of a picture to include 90% of the common issues that can arise. This approach to SLA monitoring will enable us to see at a glance what is going wrong when end user experience is sub-standard.

Posted by Chris Smith, Senior Technical Engineer