Churn prediction: from the score to the retention decision

Churn prediction is one of the better understood problems in applied analytics: the data exists, the methods are standard, and a competent team ships a workable model in weeks. Retention rates still fail to move, because the decisions that follow the score were never made. Churn was defined loosely, the highest scores were treated regardless of influence, and no control group existed. This article covers that downstream half.

Defining churn for a churn prediction model

The label is the whole problem in disguise. Two businesses using the same algorithm on the same schema will get incomparable results if they defined the target differently, and the definition is frequently inherited from whoever built the first report.

Contractual businesses have an event: a cancellation, a non-renewal, a termination request. The definition looks easy and still needs decisions. Does churn occur at the request or at the end of the notice period? Is a downgrade to a minimal tariff churn? What about a customer who cancels one of four services? Is an involuntary termination for non-payment the same outcome as a voluntary departure? It usually is not, because the intervention that helps is different.

Non-contractual businesses have no event, so churn has to be constructed from inactivity. Ninety days without a purchase means something different for a grocery retailer and for a business selling annual equipment. The window has to come from the observed repurchase distribution, and it should be chosen as a percentile of the inter-purchase interval rather than picked as a round number.

Two further decisions belong in the definition. The prediction horizon: are we predicting churn in the next 30, 60 or 90 days? A model with no stated horizon is not a model of anything specific. The observation cutoff: which data was available at the moment of prediction. Everything after the cutoff is unavailable in production, and using it in training is the most common way to produce a pilot that cannot be reproduced.

That last point deserves emphasis. A cancellation is often preceded by administrative traces: a retention note, a final-bill flag, a contract status change. Those fields are recorded around the event, so a model trained on them appears excellent and predicts nothing useful, because at scoring time they are empty. Any feature whose value is set by the churn process itself has to be excluded, and the way to find them is to check when each field is written rather than to trust the schema.

What a churn prediction score does not decide

Assume a working model. It returns, for every customer, a probability of churning within the horizon. What follows is a set of decisions the model does not make.

Which customers are worth an intervention at all. Which action to take for each. How much to spend. Who executes it and when. How the result is measured. Whether the action might make things worse.

None of these are modelling questions, and all of them determine the financial outcome. A model with modest discrimination attached to good decisions outperforms a strong model attached to none. The distinction between predicting and deciding is set out more generally in our article on predictive analytics methods; churn is where the gap between the two is widest, because the temptation to act on the ranking alone is strongest.

Churn propensity and uplift modelling

In DNA Solutions engagements, this is the error that costs the most money, and it is committed by default.

The intuitive programme takes the highest-risk decile and contacts them. But a high churn probability often means the decision is already made: the customer has bought a competitor's product, moved house, closed the business, or has a grievance no discount addresses. Spending the retention budget on the top of the ranking spends it where the probability of changing the outcome is lowest.

Worse, some interventions are counterproductive. Contacting a passive customer who had not been thinking about the contract can prompt a review that leads to cancellation. Offering a discount to someone who intended to stay converts a retained customer into a cheaper retained customer, which is a direct margin loss recorded as a success.

The quantity that matters is the change in retention probability caused by the treatment, which is an uplift question rather than a propensity one. Estimating it requires data from a randomised experiment: a population where treatment was assigned at random, so the difference in outcome between treated and untreated customers at similar risk levels can be attributed to the treatment.

This has a practical consequence for sequencing. Before an uplift model can exist, a randomised campaign has to have been run and recorded. So the first cycle of a retention programme should be built to generate that data: treat a random sample across the risk range, record who was eligible, who was selected, what was offered and what happened. The second cycle can then target on estimated uplift. Programmes that skip this step end up unable to distinguish the effect of the campaign from the effect of the selection, permanently.

Targeting rules: who to treat and at what cost

With risk, expected value and some estimate of responsiveness, the targeting rule becomes a straightforward comparison. Treat a customer where the expected margin retained exceeds the expected cost of the treatment, including the cost of treating people who would have stayed.

Customer value has to be forward-looking. Historical revenue over the last year is a poor proxy for a customer whose usage is declining, and it systematically over-values customers at the end of a contract cycle. Expected margin over a defined future window, net of servicing cost, is the right basis. Margin rather than revenue matters here: a high-revenue, low-margin customer can be worth less than a smaller one.

The action should also vary by the reason the risk is elevated, which the model can indicate through the features driving each score. Rising fault reports point to a service intervention. A tariff that has drifted out of line with usage points to a plan review, which frequently retains without a discount. Approaching contract end points to a renewal conversation. Declining engagement in a product with unused capability points to onboarding.

A discount is the default action because it is easy to execute, and it is rarely the best one, since it reduces margin on every customer who receives it whether or not it changed their decision.

A short list of guardrails belongs in the rule as well. Customers with an open complaint, those recently contacted by another campaign, and those whose risk is driven by non-payment all need to be routed differently rather than offered a retention incentive.

Measuring churn prediction with a permanent holdout

The most common measurement in retention programmes compares the retention rate of treated customers with the rate among untreated customers. That comparison is uninterpretable, because the two groups were selected differently. It usually flatters the programme, and occasionally hides a programme that is losing money.

The requirement is a randomised control group: within the population eligible for treatment, a fixed proportion is withheld at random. The difference in retention between the treated and withheld groups is the effect of the programme. Ten percent is commonly enough at reasonable volumes to detect the effect sizes that matter.

Two properties make it work. It has to be randomised within the eligible population, not a leftover group that failed some other filter. And it has to be permanent, because the effect decays: competitors change offers, the population shifts, and the treatment that worked last year stops working without announcing it. A holdout run once at launch measures a programme that no longer exists.

Expect internal resistance to withholding treatment from customers who might leave. The counter-argument is concrete: without the holdout, nobody can say whether the budget is producing retention or subsidising customers who were staying anyway, and that question will eventually be asked by someone with authority over the budget.

Alongside retention, track margin retained net of treatment cost, and the redemption pattern. Consistently high redemption of a discount offer is a signal that it is reaching people who were not at risk.

Delivering churn prediction scores to CRM and agents

A model that produces a monthly file for the analytics team has no route to action. The score has to arrive in the system where someone or something decides, at the cadence the decision occurs.

For an outbound campaign, that means the CRM segment, refreshed on the campaign cycle. For inbound service, it means the agent's screen at the moment of the call, which is the highest-value placement available: the customer has already made contact, so the intervention costs nothing extra and the timing is right. For self-service journeys, it means the cancellation flow itself.

The inbound case has a latency requirement the batch pipeline usually cannot meet. A score computed nightly is acceptable; a score computed weekly is not, because the customer's situation changed in between. Where the decision point is inside a call, the operational constraint is a lookup fast enough to render the screen, which is a serving problem rather than a modelling one. The deployment mechanics for this sit in our article on MLOps.

The agent-facing case also needs a reason alongside the score. A number with no explanation is either ignored or over-trusted. Two or three contributing factors, in plain language, with the suggested action, is what makes it usable.

What a working churn prediction programme looks like

Assembled, the pattern is: a written churn definition with a horizon and an observation cutoff; a model whose features are all verifiably available at scoring time; a first campaign cycle with randomised treatment, built to produce the data an uplift model needs; a targeting rule based on expected value net of treatment cost; differentiated actions matched to the driver of the risk; a permanent randomised holdout; and delivery of the score into the operational system at the cadence of the decision.

The model is a few weeks of that. The rest is where retention rates move.

Talk through your churn prediction programme

DNA Solutions builds these programmes end to end, from the label definition through to the serving path, as part of our predictive analytics practice. Where the customer history sits across several systems and the entity resolution has to be settled first, that groundwork runs under data and analytics, because a churn model on an unresolved customer record scores the record rather than the customer. Talk to us.