Habit: Know Your Metrics
This article is an excerpt from my longer piece Habits: Five Product Best Practices Learned at Amazon.
Know Your Metrics
Metrics, or Key Performance Indicators (KPIs), enable product owners to track the performance of a product over time.
The Learning
Why are these important?
Because PMs don’t just need to quantify everything; they also need to know how their product is performing every day. Amazon PMs dive into their metrics to see what they are telling them about customer needs. Amazon PMs can speak to the current state of their products without breaking a sweat because they know their metrics.
This is not just about using numbers for the sake of using numbers. Rather, it’s about being able to talk about how a product is performing at any moment. PMs who understand their metrics are able to quickly tell a story to anyone who asks them about the product. They are prepared to connect the dots and see opportunities in unexpected places. They can make the case for extra resources or partnership opportunities. They see problems coming before there are emergencies.
Easier Than You Think
For the uninitiated, it sounds intimidating. But after about 4 to 6 weeks of looking at product metrics closely, habit and intuition are formed.
The Amazon Systems in Support
Weekly Business Review (WBR) metrics meetings are the strictest enforcement of ‘know your metrics’ at Amazon. Your weekly callouts and operational planning documents must contain metrics or will be considered incomplete communications. You use your metrics to make the case for funding your product.
Rules of Thumb
Understand inputs (levers) and outputs that drive outcomes Amazon PMs organize metrics around things the team owns and controls. These inputs are often called “Levers”. Actively monitoring one or two company-wide metrics for context is fine, but a team needs to measure what it can control and impact.
What are inputs, outputs, and outcomes? I like to use email campaigns as a simple example of inputs and outputs.
The number of emails sent is a simple input, or lever, that you control.
The number of emails read would be an output of your email campaign.
The outcomes you are seeking might include, for example, increased brand awareness and product sales.
You can use counts and attribution models to measure your desired outcomes.
Don’t forget counter guardrail metrics. Be careful. Don’t forget to measure what the team doesn’t want to break. Do a pre-mortem to think through the unintended consequences of your push to desired outcomes. Create a guardrail metric and keep an eye on it, along with your key metrics, to make sure that no important user behaviors are being negatively impacted.
Suggested Reading
Book: Measure What Matters
Book: Winning with Data
