KPI’s for UX are always a bit of a trick because UX at its core is about people and the frustrations and pleasures they feel while using your product. Because experience is relative and hard to quantify, UX teams need to look at the quantifiable aspects of the user's journey and see how that builds into that overall experience.
The trackable aspects that we’ve identified are how long it takes (time metric), how many mistakes are made (mistake metric), can they complete the tasks (failure metric), how easy did the user think it was (ease of use metric). By calculating these together we can establish a usability metric.
Setting the groundwork
Our usability metric is based on the primary flows our users need to do within the app, these flows represent 90% of the work people are doing. Because the metric should be one that you can be evaluated as improvement over time, it’s important to have the flows focus on the primary areas of work being done. Also, in order to provide a good measure of usability, we include a good amount of edge cases within the tests. These flows are then evaluated in light of the specific metrics we introduced earlier.
New or fringe features that don’t affect the key flows of the application are evaluated separately from the usability metric.
Time
The time per task is how long in seconds it takes a user to complete a given task. That is compared to the baseline, which is the shortest time it has ever taken a user to complete the task.
Total - (seconds) Baseline / 2* = Time metric
With the time metric being an aspect that evolves as you conduct additional tests and improve the software, you’re more readily able to compare past performance to current and see the notable improvements.
Mistakes
Every time a user clicks on something that doesn’t get them towards completing the goal or are confused by the interface, then a mistake is tallied by the interviewer. These are important data points for us and notes are taken on what the mistake was and where it took place in the process.
Number of mistakes X 10* = Mistake metric
Failures
If a user reaches a point where they don’t know what to do next or cannot complete the task assigned, then a failure is registered. This is a breaking problem that indicates the highest priority UX improvement ticket and should be brought to the attention of the UX and Product lead of that product immediately.
Failure X 100* = failure metric
Ease of Use
The final part of this is the subjective experience of the users. This value is often under appreciated but subjective experience can actually be far more important than speed or mistakes.
“On a scale from 1 - 5, five being easy to use and 1 being difficult, how would you rank your this tool?”
(6 - Ease of use score ) X 200* = Ease of use metric
Usability Metric
In order to have an understandable metric that people unfamiliar with the process of developing it can easily understand, the usability score is calculated as a percentage targeting 100% usable. To get the final score, all of the metrics are added together and compared to the perfect score:
Percent usability metric = 200/(Time metric + Mistake Metric + Failure Metric + Ease of use metric)
Each of the individual metrics we discussed are numbers that you want to be trending down over time. You divide the perfect score (200) by those metrics in order to get a percentage representing how close to “perfect” usability your product is.
Final thoughts
Gathering the information and creating the KPI for UX is a really valuable exercise that enables you to track the usability over time. But alongside this, the exercise is important for us because it makes the UX team focus on the core of the application. It is often easy to get lost in the specific features you’re working on, but by regularly talking with users and evaluating the main flows, we’re able to regularly get feedback and work on the areas of the app that users are touching every single day.
*weighting of the different metrics is adjusted based on the amount of time it takes to complete a task, and the development stage of the application. Our weighting is not final
Comments