Site-Specific Deterministic Temperature and Humidity Forecasts with Explainable and Reliable Machine Learning

Core Concepts

This study investigates the feasibility of optimizing site-specific temperature and dew point forecasts by adopting the gradient boosting decision tree model XGBoost, supported by insights from Shapley Additive Explanations (SHAP) to increase the reliability of the machine learning-based forecasts.

Abstract

The study focuses on developing a framework called Multi-SiteBoost (MSB) to generate site-specific deterministic temperature and humidity forecasts using machine learning. Key highlights: The gridded numerical weather prediction (NWP) data from the IMPROVER system is used as input, along with site observation data, to train XGBoost regression models for each site and forecast variable. Various data pre-processing techniques are applied, including the inclusion of surrounding grid values, data selection, automatic feature selection, and data scaling, to optimize the model performance. The trained XGBoost models show significant improvement in forecast accuracy compared to the original IMPROVER grid values, with average reduction in hourly RMSE by 11.35% for temperature and 12.28% for dew point. SHAP analysis is used to explain the model outputs, revealing that the predictions are primarily driven by a linear combination of the main variable at different grid locations, with supplementary nonlinear effects from other features like hour, wind, and dew point. To increase the reliability of the machine learning-based forecasts, the study explores methods to identify potentially unreliable individual predictions, such as those with out-of-bound feature values. These unreliable predictions are found to have much higher error than the overall average.

Stats

The RMSE of hourly temperature forecasts from the XGBoost models is reduced by up to 1.05°C compared to the IMPROVER grid values across the selected sites. The percentage reduction in hourly RMSE ranges from 0.54% to 37.13% for temperature, and 0.54% to 37.13% for dew point, depending on the site. On average, the XGBoost models reduce the hourly RMSE by 11.35% for temperature and 12.28% for dew point, compared to IMPROVER. The percentage of critical errors (absolute error > 2°C) is reduced by 5.60% for temperature and 6.19% for dew point on average.

Quotes

"The improvement from XGBoost is found to be comparable with non-ML methods reported in literature." "The insights provided by SHAP reveal that the model outputs can be approximately described as a combination of a main component from linear combinations of selected NWP predictions and some supplementary nonlinear components."

Key Insights Distilled From

Site-specific Deterministic Temperature and Humidity Forecasts with Explainable and Reliable Machine Learning

by MengMeng Han... at arxiv.org 04-05-2024

https://arxiv.org/pdf/2404.03310.pdf

Deeper Inquiries

How can the reliability analysis methods be further improved to provide more accurate and comprehensive assessments of the model's performance?

To enhance the reliability analysis methods for a more accurate assessment of the model's performance, several strategies can be implemented: Incorporating Ensemble Methods: By utilizing ensemble methods such as bagging or boosting in conjunction with XGBoost, the model's predictions can be diversified, leading to more robust and reliable forecasts. Ensemble methods can help mitigate the impact of outliers and reduce overfitting. Feature Engineering: Conducting more in-depth feature engineering to identify and incorporate additional relevant variables that may influence the predictions. This can help capture more nuances in the data and improve the model's ability to make accurate forecasts. Dynamic Thresholding: Implementing dynamic thresholding techniques to adjust the model's confidence levels based on the specific characteristics of each prediction. This can help in identifying and flagging predictions that are likely to be less reliable. Continuous Monitoring: Establishing a system for continuous monitoring of model performance and error patterns. This can involve real-time tracking of prediction errors and adjusting the model parameters accordingly to improve reliability. Integration of External Data Sources: Incorporating external data sources, such as satellite imagery or ground-based observations, to validate and cross-check the model predictions. This can provide additional validation and enhance the reliability assessment.

How can the insights from the SHAP analysis be used to inform the development of physics-based weather models and improve their representation of local-scale processes?

The insights gained from the SHAP analysis can be instrumental in enhancing the development of physics-based weather models in the following ways: Model Validation: SHAP analysis can help in validating the outputs of physics-based models by providing a comparative analysis of the feature importance and contributions to the predictions. Discrepancies between the physics-based model and the ML model can highlight areas for improvement. Identifying Model Biases: SHAP analysis can reveal the relative importance of different features in the ML model, allowing for a comparison with the physics-based model. Discrepancies in feature importance can indicate potential biases in the physics-based model that need to be addressed. Refinement of Local Processes: By understanding the local-scale processes that contribute significantly to the predictions through SHAP analysis, developers can refine the physics-based model to better capture these nuances. This can lead to more accurate and reliable forecasts at specific locations. Enhanced Interpretability: SHAP analysis provides a clear and interpretable way to understand the model's decision-making process. This can help in identifying areas where the physics-based model may be lacking in capturing certain local phenomena, leading to targeted improvements. Model Calibration: Insights from SHAP analysis can guide the calibration of physics-based models by highlighting areas where the model may need adjustments to align with the observed data more effectively. This iterative process can lead to improved model performance and reliability.