Search papers, labs, and topics across Lattice.
This paper benchmarks LLM-based and traditional methods for log anomaly detection across four public datasets, evaluating classical parsers with ML, fine-tuned transformers, and prompt-based LLMs. Fine-tuned transformers achieve the highest F1-scores (0.96-0.99), but prompt-based LLMs exhibit strong zero-shot performance (F1: 0.82-0.91), offering a practical advantage when labeled data is limited. The study analyzes cost-accuracy trade-offs, latency, and failure modes to guide practitioners in selecting appropriate methods.
LLMs can detect system anomalies with surprisingly high accuracy even without labeled training data, rivaling fine-tuned models in scenarios where labels are scarce.
System log anomaly detection is critical for maintaining the reliability of large-scale software systems, yet traditional methods struggle with the heterogeneous and evolving nature of modern log data. Recent advances in Large Language Models (LLMs) offer promising new approaches to log understanding, but a systematic comparison of LLM-based methods against established techniques remains lacking. In this paper, we present a comprehensive benchmark study evaluating both LLM-based and traditional approaches for log anomaly detection across four widely-used public datasets: HDFS, BGL, Thunderbird, and Spirit. We evaluate three categories of methods: (1) classical log parsers (Drain, Spell, AEL) combined with machine learning classifiers, (2) fine-tuned transformer models (BERT, RoBERTa), and (3) prompt-based LLM approaches (GPT-3.5, GPT-4, LLaMA-3) in zero-shot and few-shot settings. Our experiments reveal that while fine-tuned transformers achieve the highest F1-scores (0.96-0.99), prompt-based LLMs demonstrate remarkablezero-shot capabilities (F1: 0.82-0.91) without requiring any labeled training data -- a significant advantage for real-world deployment where labeled anomalies are scarce. We further analyze the cost-accuracy trade-offs, latency characteristics, and failure modes of each approach. Our findings provide actionable guidelines for practitioners choosing log anomaly detection methods based on their specific constraints regarding accuracy, latency, cost, and label availability. All code and experimental configurations are publicly available to facilitate reproducibility.
