Our own historical replay · Snapshot 30 September 2026 · v1
An 80% forecast range does not always cover 80% of outcomes.
What Forecastall's own tests say about uncertainty, including where the model misses.
The label "80% range" describes the model's nominal interval, not a guarantee. In our historical replay, coverage changed by topic and horizon. Some intervals missed more often than their label suggests; others covered more outcomes than the target. These are Forecastall's own tests, not an independent audit.
How often did the range cover the outcome?
Six selected results, not an exhaustive ranking. Same 0-100% scale for every row.
Source: Forecastall rolling-origin backtest. Output generated . Frozen snapshot 2026-09-30-v1. Test counts are historical forecast/outcome comparisons, not independent observations.
The six results
| Topic | Horizon | Series | Tests | Inside range | Median-forecast MAE vs no-change |
|---|---|---|---|---|---|
| Stock indices | 12 months | 2 | 96 | 64.6% | -16% |
| Inflation | 2 years | 39 | 156 | 59.0% | -11% |
| Interest rates | 12 months | 3 | 145 | 68.3% | +2% |
| Currencies | 3 months | 19 | 1,082 | 83.2% | +1% |
| Commodities | 12 months | 3 | 146 | 85.6% | +1% |
| GDP growth | 3 years | 40 | 120 | 94.2% | -65% |
MAE means mean absolute error. Error comparison is a geometric mean of the per-series model/no-change MAE ratios. Negative means lower error; positive means higher error. These percentages are not investment returns. Coverage shown to one decimal; error comparisons rounded to whole percentages.
The median and the interval answer different questions
Stock-index medians had about 16% lower error than a no-change forecast in this test, while their nominal 80% intervals covered only 64.6% of 12-month outcomes. This result pools two indices; it is not a standalone result for the S&P 500. Better median error does not make an interval well calibrated.
Coverage above 80% is not automatically better: an interval can cover more outcomes by becoming wider. Compare coverage, width, median error and usefulness together. These rows have different horizons and frequencies, so they are not a league table of which topic is easiest to predict.
How the replay works
Replay dates span the last five years of each available series, spaced by 21 trading days for daily series, one month for monthly series and one year for annual series (one quarter for quarterly series). At each date the model receives only the preceding series history. It produces a median and p10-p90 range, compared with the later outcome and a baseline that repeats the last observation. Only horizons with an available later outcome enter the test.
The model is TimesFM 2.5, checkpoint google/timesfm-2.5-200m-pytorch. Minimum history is 252 daily, 36 monthly, 16 quarterly or 8 annual observations. The exact checkpoint revision, output timestamp, counts and unrounded aggregates are in the JSON download.
What this does not establish
- Tests can overlap and are not independent draws. Ninety-six tests are not 96 independent market regimes.
- Annual series are short. Source revisions may mean a replay does not use the exact data vintage available at the time.
- We have not verified a training-data cutoff for this checkpoint. It may have been exposed to historical data used in the replay. These tests do not prove performance on unseen data.
- This is a retrospective replay, not a record of forecasts published at those historical dates.
- A model using only series history cannot anticipate every policy change, war or structural break.
Reuse the snapshot
This page is a fixed snapshot, not a live scorecard. New results need a new version and date. The JSON contains all 45 family/horizon aggregates from the original output, not individual test runs or raw observations. The CSV contains only the six rows shown here. We do not publish a fabricated run-level CSV.
Snapshot source SHA-256: b1553278b2e13354696ca60952c7bab72d44237ea5f5b4ff0f74a5fd40a45acc. Download preserves the original schema and unrounded aggregate values.
Source repository: Forecastall, forecasts/backtest-summary.json and src/forecast/backtest.py. Selection: stock indices, inflation and interest rates illustrate undercoverage; currencies and commodities illustrate near/above-target coverage; GDP growth illustrates high coverage. This is a selection, not a claim that these are the six worst results.
Statistical forecasts, not investment advice. Prepared by Forecastall from its own backtest output. Explore Forecastall →