Metrics, Experiments & Reality Checks

Measuring What Matters

You can't improve what you can't measure. But measuring the wrong thing is worse than measuring nothing — it optimizes for the wrong outcome.

The Metrics Hierarchy

North Star Metric

One metric that captures the core value your product delivers. Everything else feeds into it.

Product Type	North Star Example
SaaS	Weekly active users completing core action
Marketplace	Transactions per week
Content platform	Learning hours per user per month
Dev tool	Deployments per week

Input Metrics (Leading Indicators)

These drive the north star. You can directly influence them.

Example for a learning platform:

• Course enrollment rate
• Lesson completion rate
• Return visits within 7 days

Output Metrics (Lagging Indicators)

Results of your input metrics. Harder to directly influence.

• Monthly active users
• Revenue
• NPS score

The HEART Framework (Google)

For measuring user experience:

Dimension	What it measures	Example metric
Happiness	User satisfaction	NPS, satisfaction survey
Engagement	Depth of usage	Sessions per week, time in app
Adoption	New user uptake	Signups, first-action completion
Retention	Users coming back	D7/D30 retention rate
Task success	Can users do the thing?	Completion rate, error rate, time on task

Running Experiments (A/B Tests)

Not every change needs an A/B test. But for important decisions:

1. Hypothesis: "Changing the CTA from 'Start Course' to 'Start Learning Free' will increase enrollment by 15%"
2. Metric: Enrollment click-through rate
3. Sample size: Calculate required sample (use an online calculator)
4. Duration: Run for at least 2 full weeks (account for weekly cycles)
5. Decision criteria: Define before you start — what result = ship, iterate, or kill?

Common Metrics Traps

• Vanity metrics: Total signups (without activation rate)
• Local maxima: Optimizing one metric while degrading another
• Survivor bias: Only measuring users who stuck around
• Short-term thinking: Boosting daily metrics at the expense of monthly retention
• Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure"

The Dashboard Rule

If you can't explain every number on your dashboard and what you'd do differently if it changed, you have too many metrics. Start with 3-5.

Exercise: Build Your Metrics Dashboard

For a product you work on (or this Academy platform):

1. Define the North Star metric
2. Identify 3 input metrics that drive it
3. For each input metric, describe one experiment you could run to improve it

Key Insight: The engineer who understands metrics can self-direct their work toward impact — they don't need a PM to tell them what matters.