Search papers, labs, and topics across Lattice.
This study introduces boundary-mutation testing, a novel approach for evaluating pattern-based secret scanners by varying the context in which credentials are embedded, rather than relying solely on fixed example-based fixtures. The method was applied across three different scanners, revealing that while detection rates were high in controlled contexts, significant vulnerabilities emerged when credentials were placed in specific boundary conditions, particularly with hyphen endings. Notably, two rules exhibited total failure under these conditions, highlighting the fragility of certain quantifier-based patterns and the need for robust repair methods to maintain detection accuracy without introducing false positives.
Boundary-mutation testing uncovers critical vulnerabilities in secret scanners, revealing that some rules can fail completely under realistic context variations.
Pattern-based secret scanners are commonly validated with example-based fixtures that fix one variable: the text surrounding a credential. We introduce boundary-mutation testing to vary that context, generating credentials from each rule's own regular expression, embedding them in realistic source contexts, and classifying outcomes at the rule level rather than the tool level, yielding three detection metrics. Applied to three scanners - a 43-rule open-source scanner, Gitleaks 8.21.2, and TruffleHog 3.82.13 - detection in the primary subject holds at>=0.9976 across ten contexts but collapses to 0.5233 when a credential ends in a hyphen. Five rules are affected: two, with fixed-count quantifiers, fail totally and deterministically; three, with variable-count quantifiers, backtrack and match a truncated credential; an entropy fallback rescues some failures but downgrades their severity. We validate a repair restoring full robustness with no new false positives, and report a caution: a plausible first attempt silently regressed two rules, caught only by re-running the same battery. Gitleaks has an unrelated, source-confirmed defect - a hard-coded terminator allowlist causing total misses for most credential types - while TruffleHog shows no boundary fragility but narrowest coverage. We report marginal, not conditional, failure probabilities: one common token format is structurally immune, another fails once in 64; on 292,527 lines of real code, the false-positive ordering inverts relative to the synthetic corpus. Because per-type detection is deterministic, the comparison unit is ten credential types, not hundreds of samples; no recall difference reaches significance, so we report the null result, not a ranking.
