Konsep Inti
Developing a data-efficient approach to solve diverse spatiotemporal PDEs using pretrained LLMs and cross-modal adaptation.
Abstrak
Introduction
PDEs are crucial in modeling real-world phenomena like fluid dynamics and heat transfer.
Existing solvers incur high computational costs, leading to the development of data-driven alternatives.
UPS Approach
UPS unifies different PDEs into a consistent representation space using LLMs and domain-specific neural operators.
Two-stage cross-modal adaptation process leverages pretrained LLMs and text-form meta information.
Results
UPS outperforms existing baselines on 1D and 2D datasets in PDEBench, achieving state-of-the-art results on 8 out of 10 tasks considered.
Capable of few-shot transfer to different PDE families, coefficients, and resolutions.
Methodology
Unified Data Representation: Homogenizes different PDE trajectories into a shared feature space.
Unified Architecture: Integrates FNO layers with pretrained LLMs for effective prediction.
Training Workflow
Two-stage training process: Embedding pretraining for modality alignment and task loss optimization, followed by multi-task fine-tuning on diverse PDE datasets.
Experiments
State-of-the-Art Results: UPS achieves superior performance on in-distribution tasks from PDEBench compared to existing baselines.
Generalization Studies: Demonstrates zero- and few-shot transfer capabilities to unseen PDE families, coefficients, and resolutions.
Ablation Studies
Investigates the impact of various design decisions in UPS, highlighting the importance of adapting from pretrained LLMs and incorporating metadata.
Statistik
UPSはPDEBenchの複数のデータセットで最先端のパフォーマンスを達成しました。
モデルは少ないトレーニングサンプルで強力な結果を達成しました。
RoBERTa-LargeモデルはRoBERTa-Baseよりも優れた結果を示しました。
Kutipan
"UPS outperforms existing baselines on a wide range of tasks from PDEBench."
"By adapting from pretrained models, UPS requires fewer training samples than previous approaches."