Search papers, labs, and topics across Lattice.
This paper investigates the limitations of LLMs in generating correct web API invocation code and introduces two complementary techniques鈥攔etrieval-augmented generation (RAG) and constrained decoding (CD)鈥攖o address these issues. The authors design a retriever that utilizes OpenAPI specifications for enhanced prompt injection and implement regex-based constraints to ensure compliance during code generation. Their evaluation on both synthetic and real-world datasets reveals that while RAG improves correctness in generating complete API invocations, it can lead to unnecessary parameters when endpoints are provided, whereas CD consistently enhances correctness by preventing illegal API calls and arguments.
RAG can reduce hallucinations in LLM-generated API code, but it risks introducing unnecessary parameters when endpoints are known.
Integration of web APIs is a cornerstone of modern software systems, yet writing correct web API invocation code remains challenging due to complex and evolving API specifications. Although LLMs are increasingly used for code generation, previous work has empirically shown that their ability to generate correct web API integrations is limited. At the same time, mitigation techniques and their effectiveness for this setting remain insufficiently understood. In this paper, we propose and systematically evaluate retrieval-augmented generation (RAG) and constrained decoding (CD) as two complementary approaches to improving LLM-generated web API invocation code. For RAG, we design a retriever that processes OpenAPI specifications and retrieves compact endpoint representations to inject into model prompts. For CD, we introduce an automatic translation from OpenAPI specifications to regex-based constraints enforced during generation. We evaluate both approaches on WAPIIBench's existing synthetic dataset and on a new real-world dataset derived from GitHub repositories. Our results show that RAG reduces hallucinations and improves correctness when generating full API invocations but reduces it when the endpoint is already provided as it encourages the generation of unnecessary parameters. In contrast, CD reliably prevents illegal URLs, HTTP methods, and arguments and substantially improves overall correctness for both starter codes.
