Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of combining grammar-level mutation with LLM-based code synthesis for compliance testing of PKCS#1 v1.5 signature verification across 48 cryptographic libraries. The results indicate that while the approach successfully reproduced most known specification violations and identified a new discrepancy, LLM hallucination significantly hampers effectiveness, with only 17.5% semantic fidelity despite high operational reliability. The research highlights the critical distinction between the reliability of generated scripts and their semantic accuracy, offering insights into the conditions under which LLMs can be effectively utilized in compliance testing.
LLMs exhibit a staggering 82.5% hallucination rate, revealing a crucial gap between operational reliability and semantic fidelity in compliance testing.
Testing implementations of binary protocols for specification compliance requires inputs that satisfy both structural and semantic constraints. Purely random generation and primitive mutations are often insufficient for exploring semantically meaningful behaviors in protocols that rely on Type-Length-Value (TLV) encoding, yet domain-specific compliance testing tools require deep protocol expertise and significant manual effort to construct. This work investigates whether grammar-level mutation combined with LLM-based code synthesis can serve as a viable, more generalizable approach to specification compliance testing. We evaluate the approach on PKCS#1 v1.5 signature verification -- a widely deployed TLV-encoded standard with a formally verified testing oracle (Morpheus) -- across 48 cryptographic library implementations. We reproduced 10 of 13 non-trivial specification violation categories previously identified by Morpheus, including all 5 signature forgery categories, and discovered 1 previously unreported discrepancy. We found that LLM hallucination -- occurring in 82.5% of generated scripts -- is the primary factor limiting effectiveness, not the mutation strategies. We identify five distinct hallucination types and show that their distribution varies systematically across mutation categories: structural mutations are implemented with 13.3% fidelity while constraint mutations achieve 30.3% correctness but suffer the highest rate of mutations being fully ignored (8.1%). These findings reveal a striking gap between operational reliability (99.8%) and semantic fidelity (17.5%), providing actionable guidance on when LLM-based code synthesis can be trusted in specification-driven testing pipelines.