The most common fate of a churn model is to be built, admired in a slide, and never used again. I've seen it happen with models that were technically good. The problem wasn't the maths. Nobody had decided what to do when a customer got a high score.
If you're thinking of building one, here's the order I'd do things in.
1. Write down the action first
Complete this sentence: "When the model flags a customer, we will..." It might be a call from an account manager, a discount, a prompt in the app, or a quiet note to leave them alone because they were going anyway.
This matters because the action sets the shape of the model. A call from a person is expensive, so you only want to flag the few most at-risk customers, and precision at the top of the list is what counts. An automated email is cheap, so you can cast wider. A model built for one of these can be poor for the other.
2. Define "churned" precisely
It sounds easy and it never is. For a subscription, cancelling is clear. For a shop or marketplace, customers just stop coming back, so you have to pick a rule: no purchase in 120 days, say. Look at how long customers normally wait between orders and set the window with that in mind, otherwise you'll label loyal-but-slow customers as lost.
The same applies to related problems. Early in my career I built a model to flag orders at risk of cancellation for an online marketplace, and the first job there was the same: agreeing on exactly what counted as a cancellation before touching any data.
3. Build the data as of a point in time
For each customer, pick a date in the past. Compute everything you knew about them on that day: orders, recency, support contacts, product usage. Then look forward, say 90 days, and record whether they churned.
This setup is how you avoid leakage, the quiet killer of models. If you include anything that happens after the cut-off date, like a cancellation reason or a final refund, the model will look superb in testing and fail in real use.
4. Test on a later period
Random splits flatter the model, because customers in the training and test sets come from the same time and conditions. Train on one period and test on the next. That's how you'd use it, so that's how to test it.
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np
train = df[df.snapshot_date < "2026-04-01"]
test = df[df.snapshot_date >= "2026-04-01"]
features = ["days_since_last_order", "orders_90d", "avg_order_value", "support_tickets_90d"]
model = GradientBoostingClassifier().fit(train[features], train["churned_90d"])
test = test.assign(score=model.predict_proba(test[features])[:, 1])
top = test.nlargest(int(len(test) * 0.1), "score")
print("churn rate overall :", round(test["churned_90d"].mean(), 3))
print("churn rate in top 10%:", round(top["churned_90d"].mean(), 3))
The last two lines are the ones to show people. A sentence like "among the 10 percent we flagged, 40 percent actually churned, compared with 12 percent overall" is something a manager can use. An AUC score isn't.
5. Check that the action works
A model that finds customers likely to leave doesn't tell you that your intervention will stop them. Some of those customers would have stayed anyway, and some can't be saved at any price. The way to find out is to hold some flagged customers back, contact the rest, and compare. Without that comparison you're measuring the model's hit rate and calling it business impact.
6. Plan the upkeep
Scores should land where the owner of the action already works, whether that's a CRM field, a dashboard or a Monday morning file. Re-check the model every month or two. Seasons, prices and your own product change, and the patterns shift with them.
None of this is exotic. The time goes on the definitions, the data and the follow-through, and very little on the algorithm. That's usually the case, and it's why a plain model with a clear action beats a clever one without.
If you'd like help with a first version, see our predictive modelling service.