Today I did a mock where I was reminded of the importance of developing metrics that matter to the goal and the team.

My Bias: Too many candidates jump to engagement or DAUs as their default metric. This has caused me to over-index on some indicators of quality. (Think DAU, where Activity is defined as more than just opening the app.) I also worked on optimizing Google Search, so just clicking on a link was never a good metric in my eyes, but sometimes it could be. As a mature product, clicking on a suggestion was something we looked at but would never make our decision metric. We needed to know, did they click on the link for more than a split second.

Today I was working on a thought exercise: How to improve the Apple App Store, and I found myself soo focused on quality that I overcomplicated the measure. My engineering team would have complained that I missed the point of our efforts.

Where I went wrong

I set my goal as improving search. Or improving the user’s ability to find what they are looking for, but I measured for the perfect outcome: finding the right app that you used going forward.

An Exercise for You

With that goal: “Improve Search” which metric do you think is most appropriate (or wrong) and why? (I normally emphasize the importance of three parts of the metric count, action, and time period, for this exercise I want you to focus on count and action).

(I have used the accordion format so you can see my thoughts AFTER you think through the answer for yourself).

How did you do? Do you agree with my logic? Do you have a strong counter to my suggested metrics?

In most execution interviews, being able to explain the why is half the battle. Who you are interviewing could also impact their impressions of your answer. Does your interviewer focus on working at scale, and so simple is better? Or does your interviewer spend their days focused on the tail user accounting for null results and bad experiences?

Some tips to keep in mind:

• Don’t overcomplicate your metric

• Find something close to the action you are trying to drive,

• Think about how the engineering team would feel about the metric

• Are there a number of external factors which would muddy the waters of your metric?

  • For example: If I am improving the search experience, am I responsible for apps that fail to meet their promise?

• Think about how your metric can be gamed

• Balance early actions taken with quality measures.

• Think about when initial action vs. quality action is the best metric.

  • Think downloads vs. downloads used a week later.

• What is the problem you are solving based on the user pain point you identified?

• Are you trying to improve the top or the bottom of the user funnel?

  • Measure accordingly.

• What is the stage of the product?

• What are current user trends and expectations?

Photo by Anne Nygård on Unsplash