Testing retention is harder than testing acquisition

A conversion test has a clean structure: a visitor sees one variant, converts or doesn't, within a session or two. The outcome is observable almost immediately and the unit of analysis is unambiguous.

Retention breaks all three. The outcome — did this customer still exist in month six — is not observable for six months. The unit is a cohort rather than a visit. And the customer is exposed to a continuous stream of other changes throughout, any of which could be responsible.

This is why retention experiments frequently get run as if they were conversion experiments, declared successful on a two-week readout, and quietly contradicted a quarter later.

The practical answer is to test against a leading indicator that predicts the outcome, rather than waiting for the outcome itself. Activation rate, week-four usage frequency, and payment recovery rate all resolve fast enough to test.

The step teams skip is validating the proxy. Before using week-four usage as a retention proxy, check historically whether it actually separates retained from churned cohorts. Sometimes it does strongly; sometimes it's a metric everyone assumed mattered and doesn't.

Where a proxy holds up, it's genuinely usable. Where it doesn't, using it anyway means you're optimising something with no established relationship to the thing you care about — which is worse than not testing, because it produces confident movement.

Comparing this month's churn to last month's is the most common retention analysis and among the least reliable, because the customers churning this month were acquired across many different periods under different conditions.

Cohort comparison fixes this: take everyone who joined in a given week or month, and follow that group's retention curve over time. Compare cohort to cohort rather than period to period.

This also surfaces something calendar analysis hides — that acquisition source predicts retention. Discount-acquired cohorts churn measurably faster, so a change in channel mix will move blended churn without anything about the product or lifecycle having changed at all.

Commercially, you should fix several retention problems at once. If onboarding is weak, dunning is absent and at-risk intervention doesn't exist, addressing them sequentially over three quarters costs real money in churn you could have prevented.

Experimentally, that makes attribution impossible. You will know retention improved and not which change did it, and the temptation is to credit whichever one you're proudest of.

Our position is to be explicit about which mode you're in rather than pretending. Early on, ship everything and accept you won't know the split — the business need outweighs the measurement. Once retention is roughly healthy, slow down and isolate changes, because that's when knowing which lever works starts to matter more than moving fast.

Where the volume supports it, holdout groups are the cleanest instrument — a randomly assigned segment that doesn't receive the new onboarding, the dunning sequence, or the at-risk intervention.