Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates how sequence models represent latent stochastic dynamics in a multivariate stochastic volatility setting, where only returns are observed while the true latent volatility state remains hidden. The authors identify a two-stage computation across various architectures, revealing that hidden representations effectively encode information about the next latent state, which is then mapped to return forecasts. Notably, in Transformer models, the ability to decode latent states emerges at specific architectural stages, indicating that the model's performance can be significantly affected by the alignment of output heads during training.
Hidden representations in sequence models can effectively encode latent volatility information, revealing a two-stage computation that challenges our understanding of mechanistic interpretability under partial observability.
Mechanistic interpretability has largely focused on language models and deterministic toy tasks. Much less is known about how sequence models internally represent latent stochastic dynamics under noisy, partially observed observations. We study this question in a controlled multivariate stochastic volatility setting, where models observe only returns while the ground-truth latent volatility state is known to the researcher. This setting provides a useful benchmark for mechanistic interpretability under partial observability: the latent state is hidden from the model but directly available for evaluation. Across architectures, losses, and output heads, we find evidence for a two-stage computation. Hidden representations encode substantial information about the next latent volatility state, and the output head maps this representation to squared return forecasts. Furthermore, in Transformers, latent-state decodability emerges at identifiable architectural stages whose location depends on the volatility period. In long-cycle regimes, this computation simplifies into an explicit latent-state filter consisting of a learned linear projection followed by $\ell^2$ normalization. Output-head replacement further shows that part of the degradation under noisy MSE training arises from readout misalignment rather than representation failure. These results suggest that stochastic volatility models provide a useful benchmark for mechanistic interpretability under noisy latent dynamics and partial observability.