Search papers, labs, and topics across Lattice.
The paper introduces Meerkat, a system that combines trace clustering with agentic search to detect safety violations across large sets of agent traces, addressing the limitations of per-trace judges, naive agentic auditing, and fixed monitors. Meerkat uses structured search and adaptive investigation to identify sparse failures specified in natural language, without requiring seed scenarios or exhaustive enumeration. Experiments across misuse, misalignment, and task gaming settings demonstrate that Meerkat significantly improves the detection of safety violations, uncovers widespread developer cheating, and finds more instances of reward hacking compared to previous audits.
Meerkat finds nearly 4x more examples of reward hacking on CyBench than previous audits by combining clustering with agentic search to uncover violations across many agent traces.
To identify safety violations, auditors often search over large sets of agent traces. This search is difficult because failures are often rare, complex, and sometimes even adversarially hidden and only detectable when multiple traces are analyzed together. These challenges arise in diverse settings such as misuse campaigns, covert sabotage, reward hacking, and prompt injection. Existing approaches struggle here for several reasons. Per-trace judges miss failures that only become visible across traces, naive agentic auditing does not scale to large trace collections, and fixed monitors are brittle to unanticipated behaviors. We introduce Meerkat, which combines clustering with agentic search to uncover violations specified in natural language. Through structured search and adaptive investigation of promising regions, Meerkat finds sparse failures without relying on seed scenarios, fixed workflows, or exhaustive enumeration. Across misuse, misalignment, and task gaming settings, Meerkat significantly improves detection of safety violations over baseline monitors, discovers widespread developer cheating on a top agent benchmark, and finds nearly 4x more examples of reward hacking on CyBench than previous audits.
