Search papers, labs, and topics across Lattice.
This paper introduces ReLog, a novel framework for generating logging statements for LLMs that incorporates runtime feedback to optimize for downstream debugging tasks. ReLog iteratively generates, executes, evaluates, and refines logging statements using LLMs, moving beyond static analysis and human-similarity metrics. Experiments on a Defects4J benchmark demonstrate that ReLog significantly outperforms baselines in defect localization and repair, achieving an F1 score of 0.520 and repairing 97 defects in the direct setting.
Stop optimizing LLM logs for human readability; optimizing for downstream debugging tasks like defect localization and repair yields significantly better results.
Logging statements are essential for software debugging and maintenance. However, existing approaches to automatic logging generation rely on static analysis and produce statements in a single pass without considering runtime behavior. They are also typically evaluated by similarity to developer-written logs, assuming these logs form an adequate gold standard. This assumption is increasingly limiting in the LLM era, where logs are consumed not only by developers but also by LLMs for downstream tasks. As a result, optimizing logs for human similarity does not necessarily reflect their practical utility. To address these limitations, we introduce ReLog, an iterative logging generation framework guided by runtime feedback. ReLog leverages LLMs to generate, execute, evaluate, and refine logging statements so that runtime logs better support downstream tasks. Instead of comparing against developer-written logs, we evaluate ReLog through downstream debugging tasks, including defect localization and repair. We construct a benchmark based on Defects4J under both direct and indirect debugging settings. Results show that ReLog consistently outperforms all baselines, achieving an F1 score of 0.520 and repairing 97 defects in the direct setting, and the best F1 score of 0.408 in the indirect setting where source code is unavailable. Additional experiments across multiple LLMs demonstrate the generality of the framework, while ablations confirm the importance of iterative refinement and compilation repair. Overall, our work reframes logging as a runtime-guided, task-oriented process and advocates evaluating logs by their downstream utility rather than textual similarity.
