Search papers, labs, and topics across Lattice.
This study investigates how language models encode the current year through two distinct tasks: an associative task that infers the year from verb tense and a declarative task that directly queries the year. The findings reveal that while base models can estimate the current year within one year of their training cutoff, the mechanisms for the associative and declarative tasks diverge significantly, complicating efforts to update the current year in these models. Notably, while prompting can effectively shift the declarative year, it fails to alter the associative year, which remains resistant to various update methods, highlighting a fundamental inconsistency in temporal reasoning within language models.
Language models struggle to consistently encode the current year, with associative and declarative representations diverging in their update responses.
A consistent concept of the current time is important for temporal reasoning, yet how language models represent the current time is not well understood. We contribute two tasks that probe the current year in conceptually distinct ways: an associative task, which infers the current year from verb tense, and a declarative task, which directly queries for the current year. Both tasks estimate current years within one year of the post-training data cutoff of instruction-tuned language models. For base models, predictions on the associative task serve as a strong proxy for the pre-training data cutoff, with an average error of only 10 months across 13 models. However, their internal mechanisms diverge: the associative task uses mechanisms similar to factual recall, while the declarative task lacks consistent causal pathways. This divergence poses a challenge for updating the current year in language models. None of prompting, SFT, or weight editing succeed in shifting the associative and declarative years simultaneously. Prompting updates the declarative year (94.6% success across 351 target years) but leaves the associative year nearly unchanged (1.7% success). Year-shifted SFT also fails to shift the associative year, matching the target year in only one of eight models. Weight editing, while effective for both tasks individually, does not generalize across both. Overall, our results show that the current year is not consistently encoded in language models: The associative notion, deeply ingrained in linguistic structures learned in pre-training, uses different causal mechanisms and resists the same modifications that easily shift the declarative notion learned in post-training.
Stanford HAI