Search papers, labs, and topics across Lattice.
This paper introduces Text2DSL, a formalized approach for generating domain-specific language (DSL) code from natural language descriptions, addressing the challenges of manual rule authoring in operating system security. By creating the PolkitBench dataset with 4,204 verified natural-language-to-Polkit-rule pairs and conducting controlled experiments on two models, the authors demonstrate that incorporating structured context significantly enhances code generation quality. Results show that structured context boosts syntactic validity to 98.6-99.4%, structural validity by up to 35.5 percentage points, and improves CodeBLEU scores by 60% to 95%, highlighting the importance of formal specifications in LLM-based DSL generation.
Injecting structured context into prompts can elevate DSL code generation quality to unprecedented levels, achieving near-perfect syntactic validity.
Domain-specific languages (DSLs) are widely used for managing operating system security policies, yet manually authoring rules in such languages demands high expertise and is error-prone. This paper formalises the task of automatic DSL code generation from natural language descriptions - Text2DSL - as a distinct problem class, separate from Text-to-SQL and general-purpose code generation. We introduce the PolkitBench dataset comprising 4,204 verified natural-language-to-Polkit-rule pairs, each validated through a three-level AST-based pipeline. Controlled prompt experiments on two MoE models of different scale and provenance - GigaChat-10B-A1.8B (1.8B active parameters) and Nemotron-3-Nano-30B-A3B (3B active) - demonstrate the critical role of structured context (BNF grammar, API specification, permitted identifier vocabulary) for LLM-based DSL code generation. Across both models, supplying context raises syntactic validity to 98.6-99.4%, structural validity by +9.7 to +35.5 pp, and the CodeBLEU score by +60% to +95%. The consistency of the effect across models of different scale and provenance indicates that, for the Text2DSL class of problems, injecting a formal target-language specification into the prompt context is a robust enabling factor for high-quality generation without model fine-tuning.
