FIN.18:4.4 - Compare forecasts on the information and horizon actually available

Define the forecasted quantity, observation cutoff and action horizon before measuring performance. Tomorrow’s usable bank cash, next month’s receipts and annual operating profit have different data and loss consequences. Use comparable entity and currency boundaries and the same forecast horizon. Reproduce each candidate’s stated, obtainable information basis; hold that basis constant when the claim concerns the estimator alone. Include data publication and processing delays; a value dated before the forecast origin can still have become available afterward.

Separate construction and selection from the periods used to estimate future predictive performance. Repeatedly choosing variants because they perform best on the same supposed test period uses that period for selection. Reserve a later or otherwise suitable comparison that remains outside that tuning, when the intended claim needs it. For time-dependent data, reproduce the passage of information: fit using the earlier observations, forecast the required horizon, then compare with what subsequently occurred. Repeat at suitable forecast origins rather than randomly giving a model later outcomes as training information for an earlier forecast.

Compare with a credible simple reference. A last-observation, seasonal or existing operational forecast may be useful depending on the quantity. The reference should express a plausible available continuation; a deliberately weak baseline inflates the apparent gain. Retain the old method’s actual manual adjustments and costs if they form part of how it would be used.

Examine more than one summary where the financial use needs it. Average absolute error can express typical magnitude in the same units. Signed errors reveal a tendency to overstate or understate, although offsets can hide large individual misses. Percentage errors become unstable around zero, which is common for net cash. Evaluate the times, entities and operating conditions that matter to the decision. A pooled improvement dominated by large entities can coexist with failure in the paying subsidiary.

For interval or probabilistic forecasts, compare the stated probabilities with subsequent observations over adequate comparable cases, and examine the size and location of the intervals or tails. A very wide interval may include almost everything while giving little useful funding guidance. A small test sample can expose a defect but rarely establishes stable tail probabilities. Keep the resulting claim narrower when the evidence cannot support reliability across rare shortages or changed business conditions.