Search papers, labs, and topics across Lattice.
The paper introduces MetaOthello, a suite of Othello variants with shared syntax but different rules or tokenizations, to study how transformers organize multiple world models in a shared representation space. They train small GPTs on mixed-variant data and analyze the resulting representations using causal transfer and linear probes. The key finding is that transformers trained on mixed-game data converge on a mostly shared board-state representation that transfers causally across variants, rather than partitioning capacity into isolated sub-models.
Transformers don't isolate world models for different Othello variants; instead, they learn a shared representation space that transfers causally across games, suggesting a surprising degree of generalization.
Foundation models must handle multiple generative processes, yet mechanistic interpretability largely studies capabilities in isolation; it remains unclear how a single transformer organizes multiple, potentially conflicting"world models". Previous experiments on Othello playing neural-networks test world-model learning but focus on a single game with a single set of rules. We introduce MetaOthello, a controlled suite of Othello variants with shared syntax but different rules or tokenizations, and train small GPTs on mixed-variant data to study how multiple world models are organized in a shared representation space. We find that transformers trained on mixed-game data do not partition their capacity into isolated sub-models; instead, they converge on a mostly shared board-state representation that transfers causally across variants. Linear probes trained on one variant can intervene on another's internal state with effectiveness approaching that of matched probes. For isomorphic games with token remapping, representations are equivalent up to a single orthogonal rotation that generalizes across layers. When rules partially overlap, early layers maintain game-agnostic representations while a middle layer identifies game identity, and later layers specialize. MetaOthello offers a path toward understanding not just whether transformers learn world models, but how they organize many at once.
