Every product team measures something. The question is whether they are measuring things that actually tell them whether the product is working for users, or whether they are measuring things that are easy to count and feel like progress. The difference between these two categories is larger than most people acknowledge.
Vanity Metrics and Actionable Metrics
Vanity metrics are numbers that look good but do not connect to outcomes that matter. Total registered users is a classic example — a large number is satisfying but tells you nothing about whether those users are getting value from the product. Monthly active users is better, because it tracks engagement. But even that metric needs context — active in what way, doing what, for how long?
Actionable metrics are ones that, when they move, tell you something specific about what to do next. If the percentage of users who complete onboarding drops, that is a signal to investigate the onboarding flow. If users who complete a specific action have dramatically higher retention, that is a signal to drive more users toward that action.
Defining Success Before Building
One of the most valuable exercises we do at the start of any significant feature is defining, in advance, what success looks like. If this feature works as intended, what will we see in the data? What number will move, by how much, over what time period? This exercise forces clarity about what problem the feature is solving, and it creates accountability for whether it actually solved it.
Qualitative and Quantitative Together
Numbers tell you what is happening. They rarely tell you why. A drop in conversion rate is a signal worth investigating; it does not tell you whether the drop is caused by a confusing interface, a broken form, a change in traffic quality, or something else entirely. Qualitative research — talking to users, watching session recordings, reading support tickets — is how you understand the why behind the what.
The best product teams use quantitative and qualitative signals together. They let the numbers tell them where to look and they use qualitative research to understand what they find.