Test on periods the method never saw
Hold out the last periods and forecast them one step at a time, using only earlier data each time. This is the rolling origin described in the Forecasting: Principles and Practice textbook. Errors measured on the data a method was fitted to look better than they are.
Four baselines
The mean forecasts the historical average and the naive method repeats the last value. The seasonal naive method repeats the value from the same season last cycle. Drift extends a line from the first to the last observation.
- Mean: the historical average.
- Naive: the last value.
- Seasonal naive: the same season last cycle.
- Drift: a line from first to last.
If it does not beat the baseline
The textbook says a new method that cannot improve on these simple ones is not worth considering. So the honest result is sometimes 'no improvement', with the baseline forecast as the answer. A report that always finds the fancy method best is not a test.
A language model is not evidence
A NeurIPS 2024 paper found that removing the language model from three language-model-based forecasters did not hurt and mostly helped. So do not accept a forecast because a language model made it. Ask for the backtest.
Sources and limits
- Forecasting: Principles and Practice, time series cross-validation Checked 2026-10-12.
- Time series cross-validation tests forecasts on a rolling origin: each test set is a single observation and its training set contains only earlier observations, so no future data leaks.
- Four benchmark methods are described: mean, naive, seasonal naive and drift.
- The page says forecasting methods will be compared with these simple methods and that a new method which does not improve on them is not worth considering.
- arXiv 2406.16964: Are Language Models Actually Useful for Time Series Forecasting? Checked 2026-10-12.
- Accepted to NeurIPS 2024. In ablation studies on three LLM-based forecasting methods, removing the language model did not degrade forecasting performance and in most cases the results improved.