The channel you're about to cut is probably working

Attribution is not randomly wrong. It records the touchpoints it can observe, weights the ones nearest the conversion most heavily, and cannot see anything that happened outside the window or off the device. Every one of those properties pushes credit in the same direction: toward the end of the journey.

That means demand capture — branded search, retargeting, email to people already engaged — reliably looks excellent, because it appears immediately before conversions that were already going to happen. Demand creation looks poor, because its effect is diffuse, delayed and frequently attributed to whatever the person clicked weeks later.

Once you know the bias points that way, the appropriate response is not to distrust the numbers uniformly. It is to distrust them asymmetrically: be sceptical of your best-performing channels and generous toward your worst, because that is the direction the error runs.

The pattern is consistent enough to describe. A paid search or prospecting programme reports a return well below target for several quarters. Meanwhile branded search, retargeting and email report figures that would be extraordinary if they were real. A plan forms to move budget from the first to the second.

What makes this so hard to argue against internally is that the plan is entirely rational given the reported numbers. Nobody is being careless. The dashboard says one channel returns 1.4x and another returns 9x, and moving money between them is obviously correct — unless the 9x channel is largely harvesting demand the 1.4x channel created, in which case the reallocation removes the top of the funnel and the 9x figure quietly degrades over the following two quarters.

By then the causal link is invisible. Nobody attributes the decline in branded search volume to a prospecting cut made six months earlier, so the usual conclusion is that the market got harder.

You do not need a test to know whether a cut deserves more scrutiny. Three things in data you already hold will tell you.

A geo holdout is the only method here that answers the causal question. Switch the channel off in a set of matched regions, leave it running elsewhere, run it longer than your purchase cycle, and compare total revenue rather than channel-reported revenue.

Two design details do most of the work. Run it past the cycle — a four-week test on a ten-week sales cycle measures the delay, not the effect. And pre-register what the result will change before you see it, because a holdout that produces an uncomfortable answer will otherwise get reinterpreted, and the reinterpretation is always available.

It costs real revenue in the test markets, and that cost is the point: you are buying an answer to a question that determines a much larger allocation. Run it once a year on your largest channel and you will make better budget decisions than any attribution model will give you.

Expect the result to be uncomfortable in both directions. In our experience brand campaigns frequently turn out to be far less incremental than reported, and prospecting frequently turns out to be far more. Both findings are unwelcome to somebody.

The organisational difficulty is larger than the analytical one. A holdout that shows branded search is barely incremental invalidates several quarters of reporting that people were promoted on, and the temptation to bury it is genuine. Agreeing in advance who sees the result and what it will change is not bureaucracy — it is what stops the test being expensive and then ignored.

The narrower lesson, if you take nothing else: before cutting a channel on attributed performance, check whether pausing it has ever coincided with a decline somewhere you did not expect. That question costs an afternoon and has saved more budgets than any model we have built.