Forecast accuracy, shown honestly

Walk-forward over 159 weeks out of sample. At the top aggregate level the carry-forward is marginally better; there is no number for that level.

One week backOne week on

Every week the model learns only from weeks before and is tested on exactly one week it has never seen. Then the window moves on. Over 159 weeks that gives 159 tests, none of them with knowledge from the future.

Starting point

A forecast becomes an argument only once it is measured against something. The platform from the first case study computes demand forecasts. The decision at stake is the number of animals per week. The business side asked a simple question: is the model better than what we do today? Today meant carrying the previous week forward. That carry-forward is the yardstick. It is cheap, everyone understands it, and it is harder to beat than it sounds.

I wanted an answer that holds up in front of a technical interviewer and that a managing director can read without translation. That means the same measure for model and yardstick, a period long enough to matter, and a result that also shows the uncomfortable level.

Decision

Three things were fixed before a single number was computed.

First: walk-forward instead of a single split into training and test. For every week, the model learns only from data available before that week and is then applied to exactly that week. 159 weeks in sequence, all out of sample. The test looks like production.

Second: the previous-week carry-forward as the yardstick, not a second model. If the model does not beat the carry-forward, it has no value, however elegant it is.

Third: one error measure on three levels. I use the weighted mean absolute percentage error, WMAPE for short. Large quantities weigh more than small ones, and lower is better. The decision level, animals per week, matters most. The finest level, customer by article by week, shows whether the model holds up in detail. The top aggregate level shows whether it adds anything at large.

Implementation

The model learns from sales history and is refitted every week. The carry-forward needs no training. It takes the previous week as the forecast. Both get the same 159 weeks and the same evaluation.

At the finest level that is 789,650 pairs of customer, article and week. At the decision level it is animals per week over the same period.

Alongside, I built a gate: a classifier meant to decide up front whether any demand is to be expected for a pair at all. I measured it in six variants. None was better than the simple heuristic that had done this job before. So it is out. A discarded component that was measured is worth more to me than a built-in one that merely sounds plausible.

Three more ideas I tested the same way and discarded or reverted.

Weather: the assumption was that barbecue weather drives demand. Against a calendar baseline of weekday, calendar week and year, no signal of its own remains from the weather. What looked like a weather effect was seasonality, and that is already in the model. Result: no weather interface. Holidays do matter, but they are calendar-determined and need no external source.

One shared model across all sites: sounds like more data and a better forecast. In the walk-forward it was worse than the single-site model at every site and every horizon. I reverted it and retrained the single-site models.

Article level for the animal decision: a model on the finest level, rolled up to animals, was no better than a model directly on the decision level, and it had one more error-prone mapping step. The article level stays where it is needed, in sales. For the animals, I forecast on the level where the decision is made.

Result and measurement

At the decision level the model lowers the error from 28.2 to 20.6 percent. At the finest level from 58.8 to 50.0 percent, and it does so in 150 of 159 weeks. In the remaining weeks the carry-forward was better.

At the top aggregate level the previous-week carry-forward is marginally better than the model. That belongs here, not in a footnote. The explanation is unspectacular: at large, much averages out, and there the previous week is a very good estimator. The model earns its keep one level down, where decisions are made. Anyone who only needs the total does not need the model. I tell the business side exactly that.

What I would do differently

I would build the measurement frame before the first model. Walk-forward, yardstick and the three levels are then fixed before any model goes in. Every further model gets the same test automatically, and nobody gets to pick the level on which it looks good.

I would report the uncomfortable level from the start, as a fixed part of every evaluation, not as an afterthought. An evaluation that shows only the good levels loses the trust of the business side the first time someone checks.

With the gate I should have stopped earlier. Six variants were too many. If the first two do not beat the heuristic, that usually says it all.

I would measure the range alongside the point error. An error of 20.6 percent says how far the forecast is off on average. It does not say how often a stated range held. That is the number the business side actually needs for the purchase quantity.

Technology

• LightGBM
• Python
• Walk-forward validation
• WMAPE