Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces a small transformer encoder that effectively maps Terminal-Wrench trajectories onto a unit sphere, allowing for the detection of reward hacking through a linear probe. The method achieves an AUC of 0.9467 and a TPR@5%FPR of 0.8296, outperforming a larger LLM-as-judge approach in terms of true positive rate while being significantly more cost-effective. Notably, the encoder's performance diminishes when natural-language reasoning is removed, indicating its reliance on contextual information for accurate detection.
Reward hacking detection can be achieved at a fraction of the cost with a small transformer, outperforming larger models in critical metrics.
A small transformer encoder is trained to map Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the $L_1$ distance between reward and metadata signals. A linear probe on top of that embedding detects reward hacking on the cleaned test split with AUC $0.9467$ and TPR@5%FPR $0.8296$, matching the TW sanitized LLM-as-judge AUC ($0.9510$ on the cleaned split) and exceeding its TPR@5%FPR ($0.7130$ vs $0.8296$) on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader: stripping natural-language reasoning from its input at probe time drops AUC to $0.6213$.