Search papers, labs, and topics across Lattice.
This paper introduces HECATE, a novel tool for assessing complexity in LLM-integrated applications by evaluating both prompt and code layers, addressing a critical gap in existing complexity metrics that focus solely on code. By employing a Hoare-logic-inspired formalism called Prompt-as-Specification, HECATE identifies 52 candidate metrics grounded in 25 complexity dimensions and rigorously tests them against 118 components from open-source repositories. The findings reveal that prompt-layer metrics significantly contribute to understanding complexity, with the top-performing metrics demonstrating predictive power for maintenance effort, thus highlighting the importance of prompt complexity in software behavior.
Prompt complexity is a critical dimension that significantly influences maintenance effort, challenging traditional views that prioritize code-level metrics alone.
LLM-integrated applications blend natural language prompts with program code, and much of their runtime behavior originates in the prompt layer rather than in the code itself. Existing complexity metrics, however, operate solely at the code level and therefore overlook this behavioral logic entirely. We present HECATE, the first tool designed to assess complexity in both the prompt and code layers of such applications. Central to HECATE is Prompt-as-Specification, a Hoare-logic-inspired formalism that interprets every prompt as a specification of intended behavior. Grounded in 25 complexity dimensions identified across published taxonomies, the tool generates 52 candidate metrics. We assess each metric against 118 components collected from 18 open-source repositories, relying on maintenance activity derived from version history as an empirical proxy for complexity, and discard any metric that loses significance once code size is accounted for. Only ten metrics withstand this test. Seven belong to our newly introduced set; rather than measuring sheer volume, each tallies structurally distinct elements, such as LLM call sites, memory attributes, and prompt templates, an attribute we call structural breadth. Of the three surviving conventional metrics, RFC exhibits a similar breadth-oriented character, while Halstead N and V survive only as a residual effect of size; our top-performing metrics exceed all three. Crucially, the prompt-layer metrics retain significance even when the strongest code-level metric is added as a covariate, establishing prompt complexity as a dimension in its own right. A final validation on 20 components spanning six held-out repositories shows that the two best-performing metrics continue to predict maintenance effort, supporting their generalizability beyond the training set.
