Search papers, labs, and topics across Lattice.
CelerLog addresses the latency and cost challenges of LLM-based log parsing by dynamically routing logs to either a statistical processor for common patterns or an LLM for sparse, semantically complex logs. This routing mechanism leverages the observation that most logs exhibit repetitive patterns amenable to statistical analysis, avoiding unnecessary LLM invocations. Experiments on 14 datasets demonstrate that CelerLog achieves state-of-the-art performance while being significantly faster (7.9x-18.6x compared to LLM methods) and cheaper (80.2%-94.1% token reduction).
LLMs aren't always needed: CelerLog shows you can get SOTA log parsing with a hybrid approach that's up to 18x faster and cuts token costs by 94%.
Log parsing is a fundamental step for automated log analysis, which transforms raw log messages into structured formats. Existing syntax-based parsers struggle with complex logs because they lack semantic reasoning ability. Emerging LLM-powered semantic parsers achieve high accuracy but suffer from prohibitive latency and token costs because they apply semantic inference across all logs. Our key observation is that not all logs necessitate complex semantic understanding: a vast majority of logs exhibit repetitive patterns that can be extracted via straightforward statistical analysis. Driven by this insight, we propose CelerLog, a fast and effective log parser. CelerLog introduces a dynamic routing mechanism to classify logs into dense and sparse groups. Logs with strong statistical patterns (dense groups) are processed by an efficient statistical processor, whereas the sparse groups lacking such patterns are routed to an LLM for semantic inference. This hybrid strategy avoids unnecessary LLM invocations. Extensive experiments on 14 public datasets show that CelerLog achieves leading performance over state-of-the-art baselines and is 7.9x to 18.6x faster than LLM methods and up to 1.5x faster than Drain. Additionally, it reduces costs by decreasing token consumption by 80.2% - 94.1% and LLM invocations by 86.4% - 90.9%.
