Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of Large Language Models (LLMs) in automating network intrusion detection system (NIDS) rule engineering through a user study involving 10 security experts. While LLMs generate syntactically correct rules, experts express concerns about their deployability due to issues with specificity and logic hallucinations, leading to a paradox where the generated rules are seen as partially useful. The findings highlight a preference among practitioners for using LLMs as supportive tools rather than fully autonomous rule generators, despite the favorable usability score of 67 for the system.
Experts find that while LLMs can produce syntactically valid NIDS rules, they often lack the specificity needed for real-world deployment, revealing a critical gap in trust and usability.
As network threats evolve, manual NIDS rule engineering has become a critical operational bottleneck. While Large Language Models (LLMs) show promise for automating this process, their ability to produce production-ready rules remains unvalidated. This paper presents a human-centered investigation into LLM-based NIDS rule engineering, formalizing a grounded generation framework and evaluating it through a user study with 10 domain experts. Our evaluation reveals a syntax-semantics paradox: although LLMs generate syntactically correct rules, experts find them only partially deployable due to low specificity and logic hallucinations in 12% of cases. While the system received a favorable SUS score of 67, practitioners remain skeptical of its autonomous capabilities, viewing LLMs as support tools for drafting and verification rather than independent generators. Finally, our statistical analysis indicates that while large-scale models ($\geq 70B$) consistently produce syntactically valid rules, small models ($\leq 4B$) are largely ineffective for IDS rule generation.
