Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates how instruction-tuned Transformer models, specifically LLaMA and Mistral, encode discourse relations, focusing on causation and antithesis. By framing the task as next-token prediction and employing various interpretability techniques, the authors reveal that early layers make predictive decisions mid-sequence, while mid-level layers finalize decisions closer to the end, with many layers primarily propagating earlier decisions. Notably, the findings indicate an asymmetric representation of discourse-based reasoning, where certain layers show a preference for specific answers, highlighting the nuanced internal dynamics of these models.
Early layers of Transformer models make critical predictive decisions mid-sequence, revealing a complex interplay in how causation and antithesis are encoded.
Discourse relations provide document structure, critical to language understanding and enabling language model performance and ethicality. In this work, we investigate how instruction-tuned Transformer models (LLaMA and Mistral) encode discourse relations in English, with a particular focus on the contrasting relations of causation and antithesis. Framing the task as a next-token prediction task and applying a suite of interpretability techniques to test model internals, our findings show that certain early layers make predictive decisions at mid-sequence tokens, while some mid-level layers finalize their decisions closer to the last token. Most of the remaining layers primarily propagate earlier decisions rather than actively influencing them. Additionally, we observe that some layers exhibit a preference for one answer over alternatives, suggesting asymmetric representation of discourse-based reasoning.\footnote{Our code is available at https://github.com/abhidipbhattacharyya/causation_vs_antithesis}