Summary: Quantitatively evaluate a product or service’s user experience by using metrics to gauge its relative performance against a meaningful standard.
A redesign is usually intended to produce a change in the user experience. We want to make the experience better (faster, easier, or more enjoyable). But better compared to what? Establishing a UX benchmarking practice is a great way to make sure you’re moving in the right direction and that you have a clear reference point for any improvements.
UX benchmarking refers to evaluating a product or service’s user experience by using metrics to gauge its relative performance against a meaningful standard.
We refer to benchmarking as a practice because, ideally, once you begin benchmarking, you can track your progress over time, again and again, redesign through redesign — it’s an ongoing process.
Essentially, benchmarking involves collecting quantitative data that describes the experience. For example, we might collect any of the following UX metrics:
You can collect those UX metrics using potentially any quantitative methodology, but analytics, surveys, and quantitative usability testing are the three methods that often work best for benchmarking. You can also use customer-service data (for example, the number of support emails about a specific task).
Once you have those numbers, you have to compare them to something. A single number is meaningless, since you have no idea whether it’s good or bad. With benchmarking, you can compare your UX metrics against four different possible reference points.
There’s no reason you couldn’t compare against several of these reference points. When you take your first benchmarking metrics (often called the baseline), you won’t have data from earlier versions to compare against, so it’s good to compare against a competitor or industry standard at that point.
If you work in a very niche or private industry, it may be challenging to find the exact industry standards you want. Sometimes academic institutions will publish useful data. In addition, Jeff Sauro’s team at MeasuringU.com conducts industry-standard studies for major business sectors (for example, banking, airlines, hotels, etc.).
You might be familiar with a type of UX research that helps us learn what works or doesn’t work about a design and figure out how to fix those problems. That type of research is called formative evaluation — it helps us decide how to form or shape the design. Qualitative interviews and usability testing are frequently used for this purpose.
Benchmarking is not formative; it is a summative evaluation: in other words, it helps us assess the overall performance of a (sort of) complete design (a summary of its performance). Our designs are never really complete, we’re always improving them. But you can think of a benchmarking study as a kind of snapshot in time, capturing the experience of a specific version of the product or service. Benchmarking can happen at the end of one design cycle and before the next cycle begins.