Measuring matters - by Andrew Keogh

SubscribeSign in

Measuring matters<br>But only if you're measuring the right things

Andrew Keogh<br>May 21, 2025

Share

Photo by Engin Akyurt on Pexels.comWhat Do You Measure?<br>I was working with a young company some years ago. They had some fairly typical growing pains - a lack of product direction, concern over what opportunities to prioritise, etc. Over a few weeks, we started to sort the key opportunities, get the teams moving in the right direction, and kept regularly experimenting to learn more. We were measuring progress, keeping active burndown charts and also getting regular feedback from customers. Things were moving in the right direction as far as I could tell.<br>But one of the execs was not happy.<br>I was called into a meeting to be told that he wasn’t happy with the ‘productivity’ of the team. I asked him what his concerns were.<br>‘I’ve been keeping track of when the engineers arrive at the office and none of them are ever here by 9:00 a.m.’<br>Clock-watching is not conducive to creating high-trust environments, I explained. I showed him the ways in which we were measuring progress, and how we could see that we were starting to solve the right problems for our customers. I’m not sure about how happy he was, but we never spoke about the time people arrived again.<br>‘Measure what matters’ is advice that you hear again and again in businesses of any size. Unfortunately, when it comes to software development, that often transmutes into “Measure what’s easy.” It’s easy to look at the time people arrive and leave. It’s easy to measure diffs or lines of code. Terms like ‘velocity’ have been abused to the extent that they have become meaningless, or feel like sticks to beat engineers with.<br>If you’re not thinking about ways you can reduce frictions that prevent engineers from succeeding faster, you’re probably going to end up measuring things that really don’t matter.<br>What matters to your business?

Outcomes matter. Customer experience matters.<br>Outputs are easier to measure.<br>Guess which one organisations usually focus on?<br>This isn’t to say that you shouldn’t measure outputs. Output measures are useful, but they need to be measuring progress toward outcomes. For example, if you’re trying to increase profit margins in an e-commerce store, you may want to measure number of experiments run (output) vs. profit margin per-transaction (outcome).<br>How do you decide what matters?

There are two key questions we should always ask, no matter what we are working on:<br>What outcome are we trying to achieve?

How can we tell whether we’ve succeeded?

If we can’t use actionable results or tangible customer benefits to answer these questions, then we are thinking in outputs, not outcomes. We can overcome this bias by framing hypotheses and expected results when considering potential areas for investment.<br>Don’t stop making sense

In their book Sense & Respond, Jeff Gothelf and Josh Seiden tell the story of how they worked with Taproot to deliver a new system for matching volunteers to organisations. Taproot wanted a list of features, but they eventually agreed to a contract where delivery was framed around outcomes. As a result, the team was able to build a prototype quickly and learn what features were actually required.<br>Some of the assumptions they had proved to be incorrect when tested. They saved significant time and effort not building unnecessary features. This enabled them to deliver a scalable solution that “far exceeded the performance goals written into the contract.”<br>The authors note that it’s unusual for a company to reach this sort of arrangement with an external service provider. It’s equally unusual within organisations that build their own software. Stakeholders often want tangible results in the form of features delivered so that they can demonstrate progress to the business.<br>Unfortunately, this doesn’t always translate into successfully solving customer problems.<br>OKRs to the rescue?

Objectives and Key Results (OKRs) enable organisations to frame their desired outcomes (objectives) and to state the measures or metrics they will use to assess the success of any initiative (the key results). Used well, OKRs can enhance the work of development teams, enabling them to take a data-informed approach. A key step in being successful in using OKRs or any other way of assessing organisational performance, is understanding what makes a good metric.<br>So, what makes a good metric?

In their book Lean Analytics, Alistair Croll and Benjamin Yoskovitz describe the key features of a good metric.<br>Comparability : it must demonstrate progress, e.g. a 5% increase in conversions since last month is more useful than 2% conversion

Understandability : a metric must be easily understandable by everyone in the organisation

Ratios : the most useful metrics are ratios rather than numerical.

Behavioural : a good ratio tracks customer behaviour but also will challenge the organisation’s...