Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces BT-APE, a lightweight Automatic Prompt Engineering method designed to optimize prompt construction for requirements classification tasks in large language models. By framing prompt design as an optimization problem, BT-APE utilizes LLM-generated candidates, backtracking search, and dynamic example selection, resulting in a significant reduction in computational resources while maintaining high accuracy. Evaluations show that BT-APE achieves comparable performance to the state-of-the-art PE2 method but with 72% fewer input tokens and 66% less wall-clock time, making it ideal for resource-constrained environments.
BT-APE matches the performance of a resource-heavy APE method while slashing computational costs, making it a game-changer for efficient prompt engineering in requirements classification.
Large language models (LLMs) are increasingly applied to requirements engineering (RE) tasks, yet the prompts guiding them are typically designed manually through trial and error, yielding inconsistent and suboptimal results. Automated prompt construction remains largely unexplored in RE, leaving its effectiveness unclear. To address this, we propose a lightweight Automatic Prompt Engineering approach, Backtracking APE (BT-APE), and apply it to requirements classification. We frame prompt design as an optimization problem, iteratively refining prompts via LLM-generated candidates, backtracking search, and dynamic example selection. Evaluating BT-APE on three benchmark datasets with five instruction-tuned LLMs, we compare it against four classical prompting baselines (zero-shot, few-shot, chain-of-thought, CoT+few-shot) and a state-of-the-art but resource-intensive APE baseline (PE2). BT-APE and PE2 achieve nearly identical accuracy, both substantially outperforming the classical baselines with large effect sizes; however, BT-APE imposes a far lighter computational footprint, consuming roughly 72% fewer input tokens and 66% less wall-clock time at equivalent accuracy, making it better suited to resource-constrained deployment. Our contributions are threefold: (i) a lightweight APE framework with an open interactive tool and replication package; (ii) the first systematic comparison of APE against classical prompting for requirements classification; and (iii) insights into how class definitions and prompt evolution affect performance.