Search papers, labs, and topics across Lattice.
This paper investigates the interpretability of a tokenized autoregressive Transformer agent in the Game of Hidden Rules (GOHR) by employing sparse autoencoders (SAEs) to analyze the agent's decision-making. The study reveals that SAEs can effectively recover the internal structure of the agent's decision-token embeddings, allowing for the identification of interpretable strategies and concepts that guide the agent's behavior. Key findings demonstrate that specific SAE dimensions correspond to distinct concepts and strategies, enabling a clearer understanding of how the agent navigates its environment without explicit rule labeling.
Sparse autoencoders reveal that a Transformer agent's decision-making strategies can be interpreted without explicit rule classification, uncovering hidden concepts that drive behavior.
A central challenge in interpreting learned decision-making systems is to determine whether their internal representations contain concepts that help explain their behavior. We report interpretability experiments for a tokenized autoregressive Transformer agent in the Game of Hidden Rules (GOHR). We focus on a compact two-rule task in which both hidden rules map object shapes to target buckets, but with different permutations. The policy is trained on episodes sampled from these two hidden rules and then evaluated with fixed weights. It is never given a rule label and does not use an explicit rule classifier; any rule information must be inferred implicitly from interaction history. In this setting, the correct rule is not identifiable before the agent tries an informative move and observes accept/reject feedback. Sparse autoencoders (SAEs) trained on the agent's decision-token embeddings recover this structure. When held-out decisions are labeled by simple concepts such as the chosen shape or bucket, SAE dimensions that are highly selective for a concept cover most decisions where that concept is present. Individual SAE dimensions also correspond to interpretable strategies such as probing one rule hypothesis and switching after negative feedback.
