LEMUR is a state-of-the-art log parsing framework that revolutionizes the automation of log analytics by introducing advanced techniques. It addresses challenges in traditional log parsers by utilizing entropy sampling for clustering logs and large language models for semantic comprehension. The framework achieves superior performance and efficiency in log parsing, surpassing existing methods through extensive evaluation on large-scale datasets.
Logs are essential for system monitoring, offering insights into system behavior. Log parsing separates logs into templates and variables, crucial for anomaly detection and fault diagnosis. Syntax-based and semantic-based log parsers have limitations, leading to the development of LLM-based parsers like LEMUR.
LEMUR's three key components - Information Entropy Clustering, Template Generation, and Chain-of-Thought Merging - work together to improve log analysis accuracy. The framework efficiently clusters logs based on information entropy, identifies variables using token-level analysis, and merges templates using semantic understanding from LLMs.
Extensive experiments on benchmark datasets demonstrate LEMUR's superiority in grouping accuracy (FGA) and overall accuracy (GA). The framework outperforms both supervised and unsupervised models across various datasets. Additionally, LEMUR exhibits high efficiency in execution time compared to other baseline algorithms.
The hybrid approach of Entropy + First-token sampling enhances the effectiveness of log clustering in LEMUR. The integration of Chain-of-Thought Merging further improves performance metrics like FGA and GA across multiple datasets. Overall, LEMUR stands out as a cutting-edge solution for advanced log parsing needs.
Naar een andere taal
vanuit de broninhoud
arxiv.org
Belangrijkste Inzichten Gedestilleerd Uit
by Hongcheng Gu... om arxiv.org 02-29-2024
https://arxiv.org/pdf/2402.18205.pdfDiepere vragen