Automating the Creation of Thermal Hotspot Datasets from Sentinel-2 Raw Multispectral Imagery for Onboard Artificial Intelligence

Centrala begrepp

This work presents a novel methodology to automate the creation of datasets for the detection of thermal hotspots, such as wildfires and volcanic eruptions, directly from Sentinel-2 raw multispectral data. The proposed approach leverages existing algorithms designed for processed Level-1C data to efficiently identify and annotate the corresponding raw data granules.

Sammanfattning

The paper addresses the limited availability of raw multispectral imagery datasets for onboard Artificial Intelligence (AI) applications on Earth Observation (EO) satellites. It presents a methodology to automate the creation of datasets containing thermal hotspot events, such as wildfires and volcanic eruptions, directly from Sentinel-2 raw data.

The key steps of the methodology are:

1. Procuring a list of thermal hotspot events from existing databases and online sources.
2. Downloading the corresponding Sentinel-2 raw data granules and related Level-1C products.
3. Applying a lightweight Coarse Spatial Coregistration (CSC) and Coarse Georeferencing (CG) approach to the raw data to enable the identification and annotation of the thermal hotspot events.
4. Leveraging state-of-the-art algorithms designed for Level-1C data to detect the thermal hotspots on the cropped and mosaicked Level-1C tiles, and then projecting the annotations back onto the corresponding raw data granules.

The authors showcase the application of this methodology to create the THRawS (Thermal Hotspots in raw Sentinel-2 data) dataset, which includes 1090 samples of thermal hotspots and 33,335 event-free acquisitions. The dataset and associated toolkits provide the community with a valuable resource to speed up future research on energy-efficient pre-processing algorithms and AI-based end-to-end processing systems for onboard EO applications.

Statistik

The THRawS dataset contains a total of 1090 thermal hotspot samples and 33,335 event-free acquisitions.

Citat

"To fill this gap, this work presents a novel methodology to automate the creation of datasets for the detection of target events (e.g., warm thermal hotspots) or objects (e.g., vessels) from Sentinel-2 raw data and other multispectral EO pushbroom raw imagery."
"The presented approach first processes the raw data by applying a pipeline consisting of a spatial band registration and georeferencing of the raw data pixels. Then, it detects the target events by leveraging event-specific state-of-the-art algorithms on the Level-1C products, which are mosaicked and cropped on the georeferenced correspondent raw granule area. The detected events are, finally, re-projected back on the corresponding raw images."

Viktiga insikter från

Unlocking the Use of Raw Multispectral Earth Observation Imagery for Onboard Artificial Intelligence

by Gabr... på arxiv.org 09-11-2024

https://arxiv.org/pdf/2305.11891.pdf

Djupare frågor

How can the proposed methodology be extended to create datasets for other types of target events or objects beyond thermal hotspots?

The proposed methodology for creating datasets from raw multispectral Earth Observation (EO) imagery can be adapted to detect a variety of target events or objects by modifying the event detection algorithms and the selection criteria for the bands used in the analysis. For instance, to detect different types of natural disasters such as floods, landslides, or urban development, the following steps can be taken:

Algorithm Adaptation: The event-specific algorithms currently used for thermal hotspot detection can be replaced or supplemented with algorithms tailored for the new target events. For example, for flood detection, algorithms that analyze water indices (like the Normalized Difference Water Index, NDWI) can be employed, while for urban development, change detection algorithms can be utilized.

Band Selection: Different target events may require different spectral bands for optimal detection. For instance, detecting vegetation changes might necessitate the use of near-infrared bands, while detecting water bodies may require specific shortwave infrared bands. The methodology can be adjusted to include these relevant bands in the collection used for processing.

Event Identification: The initial step of procuring a list of events can be expanded to include databases and resources specific to the new target events. This may involve integrating data from meteorological agencies, environmental monitoring systems, or satellite imagery archives that focus on the desired phenomena.

Data Processing Pipeline: The existing data processing pipeline can be modified to accommodate the unique characteristics of the new target events. This includes adjusting the spatial coregistration and georeferencing techniques to account for the specific spatial and temporal dynamics of the events being studied.

Validation and Labeling: The validation process for the new datasets should involve expert review to ensure the accuracy of event identification. This may require collaboration with domain experts in the respective fields (e.g., hydrologists for flood detection) to refine the labeling process.

By implementing these adaptations, the methodology can be effectively extended to create datasets for a wide range of target events or objects, thereby enhancing the utility of raw multispectral data for onboard AI applications.

What are the potential challenges and limitations of using raw multispectral data for onboard AI applications compared to using pre-processed Level-1C data?

Using raw multispectral data for onboard AI applications presents several challenges and limitations compared to utilizing pre-processed Level-1C data:

Data Quality and Integrity: Raw multispectral data may contain various artifacts, such as noise from the sensor, compression artifacts, and misalignment between bands due to the pushbroom acquisition mode. These issues can degrade the quality of the data and complicate the training of AI models, which typically rely on high-quality, calibrated inputs.

Increased Computational Burden: Processing raw data requires more computational resources to perform necessary corrections (e.g., radiometric and geometric corrections) that are typically handled in the ground segment for Level-1C data. This can lead to increased latency and power consumption, which are critical constraints for onboard systems, especially in small satellites.

Algorithm Complexity: The algorithms designed to process raw data may need to be more complex to account for the various non-ideal effects present in the data. This complexity can hinder the real-time processing capabilities of onboard AI systems, which are often designed to operate under strict time constraints.

Limited Availability of Training Data: The scarcity of labeled datasets derived from raw multispectral data can limit the ability to train robust AI models. In contrast, Level-1C data often come with established datasets that have been extensively validated and labeled, facilitating the development of AI applications.

Domain Shift Issues: AI models trained on pre-processed data may not perform well when applied to raw data due to differences in data distribution, calibration, and resolution. This domain shift can lead to reduced model accuracy and reliability in real-world applications.

Need for Lightweight Solutions: The necessity for lightweight processing techniques that can operate within the constraints of onboard hardware adds another layer of complexity. Developing efficient algorithms that maintain performance while minimizing resource usage is a significant challenge.

Overall, while using raw multispectral data offers the potential for more direct insights and reduced downlink requirements, it also introduces a range of challenges that must be addressed to ensure effective onboard AI applications.

How can the lightweight Coarse Spatial Coregistration and Coarse Georeferencing techniques be further improved to better handle non-ideal effects in the raw data while maintaining low computational complexity?

To enhance the lightweight Coarse Spatial Coregistration (CSC) and Coarse Georeferencing (CG) techniques for better handling of non-ideal effects in raw multispectral data while preserving low computational complexity, several strategies can be implemented:

Adaptive Shift Estimation: Instead of relying solely on pre-calculated shift values, an adaptive approach can be employed where the shift values are dynamically estimated based on the characteristics of the specific granule being processed. This could involve using a small subset of control points or features within the image to refine the coregistration process, thus improving accuracy without significantly increasing computational load.

Incorporation of Machine Learning: Machine learning techniques can be integrated into the coregistration and georeferencing processes. For instance, a lightweight neural network could be trained to predict the optimal shift values based on the input data characteristics, allowing for more precise adjustments while maintaining efficiency.

Multi-Resolution Processing: Implementing a multi-resolution approach can help manage computational complexity. By initially processing the data at a lower resolution to estimate shifts and then refining the coregistration at higher resolutions, the overall processing time can be reduced while still achieving high accuracy.

Error Compensation Techniques: Developing algorithms that can compensate for known non-ideal effects, such as sensor noise or atmospheric distortions, can enhance the robustness of the CSC and CG techniques. This could involve applying correction factors based on empirical data or using models that account for these effects during the processing stages.

Parallel Processing: Leveraging parallel processing capabilities can significantly speed up the coregistration and georeferencing tasks. By distributing the workload across multiple processing units, the overall time required for these operations can be reduced, making it feasible to implement more complex algorithms without exceeding onboard resource constraints.

User-Defined Parameters: Allowing users to define certain parameters based on the specific application or target event can enhance the flexibility and effectiveness of the CSC and CG techniques. This customization can help optimize the processing for different scenarios while keeping the computational requirements manageable.

By implementing these improvements, the lightweight CSC and CG techniques can be made more robust against non-ideal effects in raw multispectral data, thereby enhancing the overall performance of onboard AI applications while adhering to the constraints of satellite systems.