Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

Core Concepts

The authors propose TFB, an automated benchmark for comprehensive and fair evaluation of time series forecasting methods across diverse datasets and techniques.

Abstract

The authors present TFB, a comprehensive benchmark for evaluating time series forecasting (TSF) methods. TFB addresses key limitations in existing benchmarks:

Insufficient coverage of data domains: TFB includes 25 multivariate and 8,068 univariate time series datasets spanning 10 diverse domains, enabling a more thorough assessment of method performance.

Stereotype bias against traditional methods: TFB covers a wide range of methods, including statistical learning, machine learning, and deep learning approaches, to eliminate biases against certain method types.

Lack of consistent and flexible pipelines: TFB provides a unified, flexible, and scalable pipeline that ensures fair comparisons by handling dataset preprocessing, method integration, evaluation strategies, and reporting in a standardized manner.

The authors use TFB to evaluate 21 univariate and 14 multivariate TSF methods. Key findings include:

Statistical methods like VAR and LinearRegression can outperform recent deep learning methods on certain datasets.
Linear-based methods perform well on datasets with increasing trends or significant shifts.
Transformer-based methods excel on datasets with strong seasonality, nonlinear patterns, and internal similarities.
Methods considering cross-channel dependencies can significantly improve multivariate forecasting performance.
Overall, TFB enables more comprehensive and reliable evaluations, promoting progress in time series forecasting research.

Stats

Time series can exhibit diverse characteristics like seasonality, trend, stationarity, shifting, and transition.
TFB covers 25 multivariate datasets spanning 10 domains, and 8,068 univariate datasets.
Existing benchmarks have limited domain coverage, with most focusing on traffic and electricity data.

Quotes

"Time series from different domains may exhibit much more complex patterns that either combine the above characteristics or are entirely different."
"No existing MTSF benchmark has evaluated statistical methods."
"Discarding those last-batch testing samples is inappropriate unless all methods use the same strategy."

Key Insights Distilled From

TFB

by Xiangfei Qiu... at arxiv.org 04-01-2024

https://arxiv.org/pdf/2403.20150.pdf

Deeper Inquiries

How can the TFB benchmark be extended to support online/streaming time series forecasting scenarios

To extend the TFB benchmark to support online/streaming time series forecasting scenarios, several key modifications and additions can be implemented. Firstly, the data layer of the benchmark can be enhanced to handle continuous data streams by incorporating mechanisms for real-time data ingestion and processing. This would involve updating the data pool with new data points as they arrive, ensuring that the benchmark can adapt to the dynamic nature of streaming data.
Secondly, the method layer can be expanded to include algorithms specifically designed for online forecasting, such as Online ARIMA or Online LSTM. These methods are optimized for sequential data and can provide accurate predictions in real-time. By integrating these algorithms into the benchmark, users can evaluate the performance of online forecasting models under different streaming scenarios.
Additionally, the evaluation layer of the benchmark can be modified to support metrics that are relevant to online forecasting, such as Mean Absolute Percentage Error (MAPE) over sliding windows or Cumulative Prediction Error. These metrics can capture the accuracy and efficiency of online forecasting models in handling continuous data streams.
Overall, by incorporating features tailored to online/streaming time series forecasting, the TFB benchmark can provide researchers and practitioners with a comprehensive platform to evaluate and compare the performance of algorithms in dynamic and evolving data environments.

What are the potential limitations of the time series characterization approach used in TFB, and how could it be improved

While the time series characterization approach used in TFB provides valuable insights into the diverse characteristics of time series data, there are potential limitations that need to be addressed for further improvement. One limitation is the reliance on predefined characteristics (trend, seasonality, stationarity, shifting, transition, and correlation) for dataset classification. These characteristics may not capture all the nuances and complexities present in real-world time series data, leading to oversimplification and potential information loss.
To enhance the time series characterization approach, the benchmark could benefit from incorporating more advanced feature extraction techniques, such as wavelet transforms, Fourier analysis, or spectral analysis. These techniques can capture additional patterns and structures in the data that may not be captured by the existing characteristics. By expanding the feature set used for characterization, the benchmark can provide a more comprehensive and nuanced understanding of time series data.
Furthermore, the benchmark could explore the use of unsupervised learning algorithms, such as clustering or anomaly detection, to identify hidden patterns and anomalies in the data. By incorporating unsupervised techniques, the benchmark can uncover novel insights and improve the overall characterization of time series datasets.
Overall, by integrating advanced feature extraction methods and unsupervised learning techniques, the time series characterization approach in TFB can be enhanced to provide a more detailed and accurate representation of the underlying data structures.

How can the TFB benchmark be leveraged to develop meta-learning or AutoML techniques for time series forecasting

The TFB benchmark can be leveraged to develop meta-learning or AutoML techniques for time series forecasting by utilizing the diverse set of datasets, methods, and evaluation strategies available in the benchmark. Meta-learning, which focuses on learning to learn, can benefit from the rich variety of time series data in TFB to train models that can adapt and generalize across different datasets and forecasting scenarios.
One approach to incorporating meta-learning in TFB is to design meta-features that capture the characteristics of time series datasets and methods. These meta-features can be used to train meta-learners that can select and optimize forecasting models based on the specific properties of a given dataset. By leveraging the extensive dataset collection in TFB, meta-learning algorithms can learn to identify patterns and relationships that lead to improved forecasting performance across diverse domains.
Similarly, AutoML techniques can be developed using TFB to automate the process of model selection, hyperparameter tuning, and feature engineering for time series forecasting. By integrating AutoML capabilities into the benchmark, users can streamline the process of designing and evaluating forecasting models, ultimately leading to more efficient and effective forecasting solutions.
Overall, by integrating meta-learning and AutoML techniques into the TFB benchmark, researchers and practitioners can explore innovative approaches to optimizing time series forecasting models and advancing the state-of-the-art in the field.