GAS-Norm: Improving Deep Learning Time Series Forecasting by Adaptively Normalizing Non-Stationary Data

Core Concepts

Deep learning models for time series forecasting often struggle with non-stationary data; GAS-Norm improves performance by combining a Generalized Autoregressive Score (GAS) model with deep neural networks to adaptively normalize input data and denormalize predictions.

Abstract

GAS-Norm: Score-Driven Adaptive Normalization for Non-Stationary Time Series Forecasting in Deep Learning (Research Paper Summary)

Bibliographic Information: Urettini, E., Atzeni, D., Ramjattan, R. J., & Carta, A. (2024). GAS-Norm: Score-Driven Adaptive Normalization for Non-Stationary Time Series Forecasting in Deep Learning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM ’24), October 21–25, 2024, Boise, ID, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3627673.3679822

Research Objective: This paper investigates the challenge of applying deep neural networks (DNNs) to non-stationary time series forecasting and proposes a novel normalization method, GAS-Norm, to improve their performance.

Methodology: The authors first demonstrate the vulnerability of DNNs to non-stationary data through a simple experiment with a Lorenz attractor. They then introduce GAS-Norm, which combines a Generalized Autoregressive Score (GAS) model with DNNs. The GAS model filters and predicts time-varying means and variances, enabling adaptive normalization of the input data and denormalization of the DNN's predictions. The authors evaluate GAS-Norm's performance against other normalization techniques on synthetic and real-world datasets using various DNN architectures.

Key Findings: GAS-Norm consistently outperforms other normalization methods, including Global Norm, Local Norm, Batch Normalization, and RevIN, across various datasets and DNN architectures. The method proves particularly effective in handling non-stationary data with changing means and variances, leading to more accurate forecasts.

Main Conclusions: GAS-Norm effectively addresses the limitations of DNNs in handling non-stationary time series data by providing adaptive normalization and denormalization capabilities. This approach leverages the strengths of both statistical modeling (GAS) and deep learning, resulting in improved forecasting accuracy.

Significance: This research contributes a novel and effective method for improving the performance of deep learning models in time series forecasting, particularly in challenging non-stationary environments.

Limitations and Future Research: While GAS-Norm demonstrates significant improvements, the authors acknowledge that further enhancements are possible. Future research could explore incorporating seasonal GAS models and tailoring distributional assumptions for specific data characteristics.

Stats

Deep forecasting models improve their performance in 21 out of 25 settings when combined with GAS-Norm compared to other normalization methods.
GAS-Norm normalization for NN5 weekly, M4 weekly, and Fred MD took 4.08s, 28.46s, and 8.37s, respectively, at a mean and variance strength of 0.
The same datasets with a strength of 0.5 yielded runtimes of 14.55s, 61.08s, and 307.02s, respectively.

Quotes

"Despite their popularity, deep neural networks (DNNs) applied to time series forecasting often fail to beat simpler statistical models."
"In this work, we propose GAS-Norm, a novel normalizing approach that combines DNNs and Generalized Autoregressive Score (GAS) models, a class of statistical autoregressive models developed to handle time series data with time-varying parameters."

Key Insights Distilled From

GAS-Norm: Score-Driven Adaptive Normalization for Non-Stationary Time Series Forecasting in Deep Learning

by Edoardo Uret... at arxiv.org 10-08-2024

https://arxiv.org/pdf/2410.03935.pdf

Deeper Inquiries

How might GAS-Norm be adapted for multivariate time series forecasting, where the relationships between different variables are also non-stationary?

Adapting GAS-Norm for multivariate time series forecasting with non-stationary relationships between variables requires moving beyond the univariate GAS model used for individual features. Here's a potential approach:

Multivariate GAS Model: Instead of modeling each feature's mean and variance independently, employ a multivariate GAS model. This model would estimate a time-varying mean vector and covariance matrix for the entire input vector at each time step. Examples include the multivariate GARCH models (e.g., DCC-GARCH, BEKK-GARCH) or GAS models with multivariate distributions like the multivariate Student's t.

Normalization with Covariance: Utilize the estimated time-varying covariance matrix to normalize the input data. This could involve techniques like whitening transformation, which decorrelates the input features and scales them to have unit variance. This step ensures that the relationships between variables are also normalized, accounting for their changing dynamics.

Denormalization: Similar to the univariate case, denormalize the DNN's output using the forecasted mean vector and covariance matrix from the multivariate GAS model. This step reconstructs the original scale and correlations of the predicted variables.

Challenges: Implementing a multivariate GAS-Norm introduces challenges such as:

Computational Complexity: Multivariate GAS models are computationally more demanding than their univariate counterparts.
Model Selection: Choosing an appropriate multivariate GAS model and its specification (e.g., number of lags) becomes crucial.
Overfitting: With more parameters to estimate, overfitting to the training data becomes a concern, especially with limited data.

Could the reliance on a pre-defined parametric distribution for the GAS model limit the flexibility of GAS-Norm in capturing complex non-stationary patterns?

Yes, relying on a pre-defined parametric distribution for the GAS model could limit GAS-Norm's flexibility in capturing complex non-stationary patterns. Here's why:

Distribution Misspecification: If the true underlying distribution of the data deviates significantly from the assumed parametric distribution (e.g., Gaussian, Student's t), the GAS model may not accurately capture the time-varying parameters. This misspecification can lead to suboptimal normalization and, consequently, affect the DNN's forecasting performance.

Limited Flexibility: Parametric distributions impose assumptions about the shape and characteristics of the data. While they offer computational efficiency, they might not be flexible enough to represent complex non-stationary patterns, such as those with time-varying skewness, kurtosis, or multimodality.
Addressing the Limitations:

Flexible Distributions: Explore GAS models with more flexible distributions, such as skewed t-distributions or mixture models, to accommodate deviations from normality and capture more complex patterns.

Non-parametric Approaches: Consider non-parametric methods for estimating the time-varying parameters, such as kernel density estimation or quantile regression. These methods make fewer assumptions about the underlying distribution but can be computationally more intensive.

Hybrid Models: Combine parametric and non-parametric approaches. For instance, use a parametric GAS model to capture the overall trend and a non-parametric method to model the residuals, capturing finer-grained non-stationary behavior.

If deep learning models struggle with relatively simple non-stationary patterns, does this suggest limitations in their ability to learn complex temporal dependencies, and how might these limitations be addressed?

The struggles of deep learning models with simple non-stationary patterns do suggest potential limitations in learning complex temporal dependencies, but these limitations are not insurmountable. Here's a breakdown:
Limitations:

Overfitting to Training Distribution: DNNs excel at recognizing patterns within the distribution of their training data. Non-stationarity shifts the data distribution over time, making it harder for the model to generalize to unseen patterns.

Difficulty in Extrapolation: DNNs often struggle to extrapolate beyond the range of observed data. Non-stationarity introduces trends and shifts that might require the model to extrapolate, leading to poor performance.

Vanishing/Exploding Gradients: Recurrent architectures, commonly used for time series, can suffer from vanishing or exploding gradients, especially with long sequences. Non-stationarity can exacerbate this issue by introducing large variations in the input.
Addressing the Limitations:

Adaptive Learning Rates: Employ adaptive learning rate methods like Adam or RMSprop to adjust the learning rate during training, improving convergence and handling non-stationarity.

Regularization Techniques: Utilize regularization techniques like dropout, weight decay, or early stopping to prevent overfitting and improve generalization to unseen data patterns.

Attention Mechanisms: Incorporate attention mechanisms, as seen in Transformers, to focus on relevant parts of the input sequence and capture long-range dependencies more effectively.

Hybrid Models: Combine DNNs with traditional time series methods. For example, use a statistical model to capture the non-stationary trend and a DNN to model the residuals, leveraging the strengths of both approaches.

Curriculum Learning: Train the model on increasingly complex non-stationary patterns, gradually exposing it to more challenging data distributions.

Data Augmentation: Generate synthetic time series with varying non-stationary properties to augment the training data and improve the model's robustness to distribution shifts.