Spotting orders that were likely to be cancelled, before they were

The situation

On an online marketplace, an order is a promise between a buyer and a seller. When that promise breaks and the order is cancelled, everyone loses a little: the buyer is let down, the seller has wasted effort, and the marketplace has done a lot of work for nothing.

The problem

Cancellations were reported after they happened. The team could see last week's rate but had no way to tell which of today's orders were in trouble. Support and operations teams could only react once the damage was done.

What we did

The first job was a plain conversation: what exactly counts as a cancellation? Cancelled by the buyer, by the seller, or by the platform? Before or after the item was packed? Agreeing on that one definition mattered more than any technique that came later.

Then a model learned from past orders. It looked at what was known at the moment an order was placed, such as the type of item, the seller's track record and how the buyer had behaved before, and gave each new order a risk score. We were strict about one rule: the model could only use information that existed at that time. Anything learned later, like a cancellation reason, would have made the results look wonderful in testing and useless in real life.

The scores were then tested on a later period the model had never seen, which is the honest way to check whether it works on tomorrow's orders and not just yesterday's.

The outcome

The team got a ranked view of orders by risk, so attention could go to the few that needed it first. A seller could be nudged, a buyer could be contacted, or an order could be watched more closely, all before it fell through.

What it meant for the business

A number that used to arrive as bad news after the event became something the team could act on while there was still time. It also gave everyone one shared meaning of "cancelled", which cleaned up the reporting too.

Could this work for you?

If you lose revenue to customers who leave, orders that fall through, or leads that go quiet, the same approach applies. The model is the smaller part. The bigger part is agreeing what counts as the event and what your team will do when the score is high.