Search papers, labs, and topics across Lattice.
This paper investigates the impact of LLM involvement on chatbot accuracy in static code analysis by comparing three architectures: direct query generation, schema-constrained JSON intermediate representation, and tool-augmented agentic generation. They evaluate these architectures on a benchmark of 20 code analysis tasks using four open-weight models. The results show that a structured intermediate representation achieves the highest result match rates, outperforming direct generation and agentic approaches, especially with larger models.
LLMs are better at code analysis when forced to output structured data, beating agentic approaches while using 8x fewer tokens.
Large language models are increasingly used to make static analysis tools accessible through natural language, yet existing systems differ in how much they delegate to the LLM without treating the degree of delegation as an independent variable. We compare three architectures along a spectrum of LLM involvement for translating natural language to Joern's query language \cpgql{}: direct query generation (\approach{1}), generation of a schema-constrained JSON intermediate representation (\approach{2}), and tool-augmented agentic generation (\approach{3}). These are evaluated on a benchmark of 20 code analysis tasks across three complexity tiers, using four open-weight models in a 2\(\times\)2 design (two model families \(\times\) two scales), each with three repetitions. The structured intermediate representation (\approach{2}) achieves the highest result match rates, outperforming direct generation by 15--25 percentage points on large models and surpassing the agentic approach despite the latter consuming 8\(\times\) more tokens. The benefit of structured intermediates is most pronounced for large models; for small models, schema compliance becomes the bottleneck. These findings suggest that in formally structured domains, constraining the LLM's output to a well-typed intermediate representation and delegating query construction to deterministic code yields better results than either unconstrained generation or iterative tool use.
