Does anyone else attempt to answer this question? What do you measure and how?
I’m a new QA Manager of an ecommerce product. My boss has challenged me with determining if the quality of the product-under-test is going up or down (in production) per quarter. I’m a metric hater and really struggling with how to answer this reasonable question.
• My DevOps intuitions take me to the four DORA metrics, but those are more about throughput and stability than product quality.
• My other fallback is bugs found in production (escapes). But I kind of hate that one too b/c as the product grows offering more value (quality), that growing value-to-users may outpace increasing escape amounts.
• This is a global customer facing ecommerce site so user feedback seems like a good quality measure. I’m not sure if we collect it. Surely we must. I’ll ask. Active users per time period. I can get that. Will these two user-facing measures suffice?
What am I missing?
“Quality is value to some person (who matters)” - Jerry Weinberg’s quote adapted by Michael Bolton & James Bach.
Who are your stakeholders? Who is asking about quality? What is it to them? This question is one of the most important starters when you get asked to talk about quality. I’ve added some question suggestions to define on what quality is to you in my article what is quality 1.
I love metrics, but I hate the misleading, confusing, demotivating metrics. There’s a lot of those, unfortunately. Like the bug number. It can be rigged. How motivating it is as well? If I don’t report anything, does it make the product better quality? Not really.
I am a big fan of team productivity metrics like Accelerate’s 4 key metrics 2. As well as using monitoring for the agreed upon success KPIs/North Star metrics by the company. Working closely with product on defining those metrics can be super motivating for the whole company.
For example, you could have a new feature that is supposed to increase the revenue (simplified example, likely it won’t increase the revenue directly, but should increase engagement which also will increase the revenue). So, let’s measure once we release it. If it turned out to be true - it’s a success.
Going on pure sales figures is still a decent measure, but has it’s problems as not being a honest metric.
My “anti rose-tinted-spectacles” take however, is that any attempt to measure merely changes the thing being measured. At the end of the day designing a product so that it becomes consumed like a SaaS. I also worry that it might even happen that you do all of the righty things, and a competitor comes along with a better advertising campaign and kills the company.
So many things change in our industry so often, than nothing technical more than “is it making money still?” makes sense. Obviously security risks are the ONE thing, and no amount of KPI or North Star will cover you at all. At the end of the day the business should know if they are happy with quality based on their specific risk profile and needs in the current climate.