Metrics Loop: Product-Driven Approach to KPIs

When defining product KPIs, starting with a 'driver tree' approach is always a good idea, but if you find yourself questioning metric relationships - this alternative approach can help.

Just last week

Lenny Rachitsky

and

Dan Hockenmaier

have published a great piece about breaking down the key business metrics of your product into an equation. I highly recommend all product analytics folks to try to take this approach through its paces for your own product. Especially if you are new to the company/product or the opposite - if you are drowning in entirely too many dashboards.

From chaos to order

The idea is simple - you break the main business outcome (almost always a version of revenue) into its direct arithmetical inputs, and then try to break every component of the equation into its own formula.

For example, for a subscription SaaS product with different price tiers (i.e. ability for customers to upgrade/downgrade), the first-level formula will be:

Annual Recurring Revenue (ARR) in current period = New ARR (i.e. revenue from first-time customers) + Reactivation ARR + Expansion ARR (i.e. incremental revenue from upgrading customers) - Contraction ARR (i.e. incremental decrease in revenue from downgrades) - Churned ARR 

For the second-level formula, we would take each component and break it down further, e.g.:

New ARR = Web traffic x Conversion Rate to Signup x Conversion Rate from Signup to Trial x Conversion from Trial to Paid Subscription

.. and so on.

So we are more or less landing on a metrics tree:

Layering In the Context

This already goes a long way in so many organizations that aren’t used to thinking this way and usually focus on just high-level trends of ARR, Net Revenue Retention, Customer Acquisition Cost (CAC) and try to guess where the trend changes come from.

But, in my experience, this is also where the complications begin - especially on the product analytics side. We can already see from this tree that the New ARR inputs already feel pretty actionable, while, for example, Churn or Reactivation ARR inputs feel very abstract.. Churn rate is also a pretty lagging metric - we probably will need to start looking at some product engagement? Same for Expansion and Contraction - they will likely be tied to a version of product engagement too?

But then our ‘arithmetics’-based approach somewhat breaks.

There is a lag between engagement change and when it manifests in a revenue ‘event’. If we’re talking about subscription tiers, the engagement change needs to be of a big enough magnitude to warrant a revenue ‘event’. How do we define metrics outside of a pure formulaic approach that relate to the product experience, and still preserve ‘leading to lagging’ relationship or maybe event causality?

At Spring Health, a mental health employee benefit startup, we ran into another complication. We, of course, cared a ton about delivering good clinical outcomes to our end users. But to some extent it went against the revenue formula - because there was a usage-based component to revenue - payment per therapy appointment. If someone has great outcomes quickly - then they don’t need to attend any more sessions, so are we losing money?

Of course not, because if we get a customer’s employee population to be healthier, the customer will be more likely to renew with Spring, refer Spring to other companies, they may achieve some savings too - which can enable them to hire more employees generating revenue for Spring.

You probably see where I’m going with this, the clinical outcomes measure - one of the most important indicators for the company - didn’t quite fit so neatly in a metrics tree or a formula…

Growth Loop for KPI Organization - Yes, Please!

… and this is how the ‘flywheel’ or growth loop KPI framework approach was born, in an attempt to create an organized set of metrics around product health and engagement that reflects both short- and long-term sustainable growth for the company.

Here is how it works:

• Map out a qualitative growth loop of the product - in other words, how does a new user transition from initial product engagement to generating initial revenue, retained revenue (the opposite of churn), expanded revenue in the short term, and help the revenue grow long term (e.g. refer friends, invite coworkers, share on social media)

• For each ‘stage’ or ‘transition of the loop - define (qualitatively, for now) - what are the 2-3 key ‘enablers’ and ‘constraints’ of this transition:

  • Constraints: what are some aspects of product that make this transition challenging to improve/optimize, what puts it at risk?

  • Enablers: what is something special/differentiating about the product that makes this transition successful and defensible?

• Now, put your data to work:

  • Try to define a metric for each loop transition, enabler, and constraint

  • Try to tease out/prove correlation (or causation - if you can experiment against these metrics) - both between enablers/constraints and the loop transition metric, and between loop transition metrics and business outcome.

Let’s look at an example…

Let’s say our product offers executive coaching. We algorithmically match vetted coaches to specific challenges a customer is facing and help with effective resolution. Given our customer base, we rely on referrals and positive reviews quite a bit for our growth, and we aim to be able to solve a variety of challenges, so our customers start engaging with us consistently for ongoing support - versus for a specific one-off challenging situation. Let’s assume that the revenue model is usage-based (pay per appointment).

Our simplified growth loop can look something like this:

It has four key transitions:

• Activation (signup → first session)

• Repeat Engagement (first session → returning session)

• Effectiveness/Outcome (several sessions → resolved executive challenge)

• Word of Mouth (technically two transitions here: outcome → making a referral; referral → redemption)

From the brief prompt above and making some assumptions about our fictional product, we can tease out enablers and constraints. It mentioned ‘matching’ and ‘solving a variety of challenges’, so our enablers can be related to some unique ML algorithm and/or great variety in coaching supply, but to match someone - they will likely need to fill out a bunch of intake, which can be a constraint for a busy executive. For repeat engagement, the coach will need to be a good fit and make a great impression on the first session…

With this, our loop with enablers and constraints can look something like this:

And now, the only thing that is left is for each ‘colorful box’ on the map to create a metric that makes sense from the product standpoint.

Let’s take the ‘Activation’ transition as an example, here is what it could look like:

• Activation: % of new customers attending a coaching session within 14 days of signup

• Enabler - Coach Availability: ‘% of new customers viewing 2+ coaching with at least one available appointment within 14 days’

• Constraint - Intake Completion: ‘% new customers completing the intake survey within 24 hours of signup’

Of course, the real application even for our coaching product will need some more work - given that it sounds like a marketplace product and a product with high COGs (coaches need to be paid for their services!) - it would make sense to also look at the supply side of things (what reflects a good experience for a coach and ensures the variety/availability of coaches?) and network effects between supply and demand.

Now, Your Turn

Interested in trying to replicate this approach? Try it our for your product and report back! Did you find any metrics that weren’t on your original metrics tree?

For more practice with both metrics trees and loops, as well as North Star Metric framing, and key analytics techniques for metric refinement - join my upcoming Product Analytics cohort on Uplimit!

For now, here are the steps to follow:

1. Map out all the ‘actors’ in your product ecosystem, reflect on the problems your product is solving for them, and the dynamics of their relationships. Whose value scales with revenue? Who pays the bills? Who is the cornerstone actor - if they are underserved the ecosystem breaks?

2. Reflect on the long-term growth of the business over the span of several years. How does your revenue expand? To what extent are you constrained by serving and acquisition cost?

3. With #1 and #2 in mind, try to map out a core qualitative growth loop of your product. If your product is a marketplace or a B2B product, it would likely be somewhat of a simplification of the reality (a system of loops), but a much needed one at this stage.

4. Try to describe each transition of the loop with a qualitative concept and come up with a metric describing how well the transition works (remember, perfect ideas imperfectly measured are better than perfect measures for imperfect ideas, but if you can - enlist a data or a finance person’s help)

5. Come together with your cross-functional leads and brainstorm (qualitatively) 2-3 enablers and constraints for each loop transition. For each enabler or constraint - pick a metric.

6. With a data person’s help, capture your metrics (and their hierarchy/connections) on a dashboard with enough visual context to gauge weekly/monthly and year over year movements for each metric

Put it through its paces! Every week try to reflect on the movements of higher-level metrics in relation to lower-level metrics to test the framework for completeness and redundancy.