Search papers, labs, and topics across Lattice.
This paper introduces a two-level internal readout for mixture-of-experts (MoE) reasoning, distilling a complex vocabulary-scale representation into a 64-axis semantic frame (J64) that captures the model's reasoning states. The authors demonstrate that J64 enhances predictive performance significantly, achieving an AUC improvement of 0.096 to 0.135 over baseline methods, while a low-overhead proxy (R64) maintains 95 to 100% of J64's predictive gain with median correlations of 0.69 to 0.86. Additionally, the readout mechanism supports improved decision-making during generation and routing, shifting reasoning behaviors from numerical guessing to exact symbolic execution.
J64 reveals hidden reasoning states that can significantly boost model accuracy and decision-making, while R64 provides a lightweight, effective proxy for deployment.
What a reasoning model writes is only a partial record of the process that produces it. We introduce a two-level internal readout for mixture-of-experts reasoning. We first distill vocabulary-scale J-space into J64, a 64-axis semantic frame learned from the model's own reasoning states. J64 reveals readable process state that the emitted trace does not show: it separates inference effort from problem-induced strain. It also adds 0.096 to 0.135 held-out AUC over a baseline that reads the same rollout as token occupancy and aggregates it in exactly the same way. We then reconstruct J64 from native expert-routing statistics. The result is R64, a low-overhead proxy: its median per-axis correlation with J64 is 0.69 to 0.86 across three models and two families, and on gpt-oss-20b it preserves 95 to 100% of J64's predictive gain. The readout supports test-time decisions at two temporal resolutions. Over completed candidate sets, J64 and R64 improve single-branch selection, and R64-weighted voting improves plain majority voting in seven of eight settings. During generation, rolling readout windows drive a cumulative stop-and-resample policy whose operating point is fixed on training questions alone. J64 improves accuracy by 1.1 to 5.9 points over a sibling-permuted control, and the routing-only R64 proxy retains 0.9 to 3.2 of those points. Finally, router edits aimed at the mechanism J64 names induce the predicted reasoning behaviors and shift a diagnosed stall from numerical guessing toward exact symbolic execution. Together, J64 makes latent process state readable, while routing makes it deployable and actionable.
