Search papers, labs, and topics across Lattice.
This study investigates the performance of Agent Skills in code auditing tasks under varying context lengths, revealing that while Codex with gpt-5.4-mini achieves high requirement coverage, it suffers significant performance drops in longer contexts. Specifically, it passes 8 out of 10 runs in a clean context but only 3 out of 10 in both relevant and irrelevant long contexts, indicating that even minor omissions can lead to substantial failures. The findings highlight the limitations of generic self-checks compared to detailed external checklists, which perform significantly better, suggesting that context length impacts the reliability of procedural instructions in agent workflows.
Codex's performance plummets by 50% in long contexts, revealing critical vulnerabilities in agent skills during code audits.
Agent Skills package procedural instructions and checks for use by general-purpose agents, but loading a skill does not guarantee that every requirement remains active throughout a long tool-using trajectory. We study this problem in a production-derived, white-box code-audit workflow. Holding the task and 24 artifact checks fixed, we vary the surrounding context and classify where failures first become visible: lost requirements, editing drift, failed checking, or non-agent evaluator/runtime failures. Codex with gpt-5.4-mini passes 8/10 runs in a 10,991-character clean context but only 3/10 in both a 299,140-character relevant context and an equal-length irrelevant context. This 50-percentage-point difference is large but remains trend-level under two-sided Fisher tests (p = 0.0698). Requirement coverage nevertheless stays above 92% in both long conditions, showing that a few omissions can invalidate an otherwise complete artifact. A second task passes all clean and long runs, so the evidence does not support a universal context-length threshold. A detailed external checklist passes 10/10 runs, compared with 5/10 for a generic self-check (p = 0.0325). Coding-agent scaffolds may help by selecting a smaller working set, but they do not eliminate failures. We do not introduce context rot or a new general monitoring method; we provide a bounded failure classification and empirical case study for white-box code auditing.
