EarthGPT: A Universal Multi-modal Large Language Model for Remote Sensing Image Comprehension

Concepts de base

EarthGPT is a pioneering multi-modal language model designed for remote sensing image comprehension, offering superior performance in various tasks and demonstrating robust generalization capabilities.

Résumé

EarthGPT is a universal multi-modal language model developed for remote sensing image comprehension. It integrates various RS interpretation tasks, including scene classification, image captioning, visual question answering, and object detection. The model proposes a visual-enhanced perception mechanism to refine and incorporate semantic information at different scales. Additionally, it introduces a cross-modal mutual comprehension approach to deepen the understanding of both visual and language content. EarthGPT also presents a unified instruction tuning method for multi-sensor tasks in the RS domain. The MMRS-1M dataset is constructed to address the lack of expertise in MLLMs for RS images. Extensive experiments show EarthGPT's superior performance compared to specialist models and MLLMs in various RS tasks.

Stats

MMRS-1M dataset comprises over 1M image-text pairs based on 34 existing diverse RS datasets.
EarthGPT achieves 77.37% accuracy on the CLRS dataset and 74.72% accuracy on the NaSC-TG2 dataset.

Citations

"EarthGPT offers a versatile paradigm for open-set reasoning tasks."
"Our code and dataset are available at https://github.com/wivizhang/EarthGPT."

Idées clés tirées de

EarthGPT

by Wei Zhang,Mi... à arxiv.org 03-11-2024

https://arxiv.org/pdf/2401.16822.pdf

Questions plus approfondies

EarthGPTの性能は、他のMLLMと比較して実世界のシナリオでどうですか？

EarthGPTは、多様なリモートセンシングタスクにおいて優れたパフォーマンスを発揮します。特に、画像分類、画像キャプショニング、ビジュアル質問応答（VQA）、視覚グラウンディング、物体検出などのタスクにおいて他の専門モデルやオープンセットモデルよりも優れた結果を示しています。これはMMRS-1Mという大規模で包括的なデータセットを活用し、リモートセンシング領域でのマルチモーダル対話型アシスタントとして高度な能力を獲得したことに起因します。