Search papers, labs, and topics across Lattice.
This study investigates the internal encoding of type information in state-of-the-art code models by probing their hidden states with a dataset of Java and Python examples. The findings reveal that even untyped code can yield cross-lingual type representations, and that hidden states can linearly encode result types across languages. Additionally, the robustness of these representations to lexical perturbations and syntactic variations is demonstrated, filling a significant gap in the interpretability of code models related to formal type semantics.
Cross-lingual type representations can be extracted from untyped code, revealing hidden structures in state-of-the-art code models that challenge our understanding of their internal workings.
State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.
