Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
We introduce the Prior-Fitted Language Model (PFLM), a 300M-parameter transformer pretrained on synthetic data generated by a distribution of recurrent structural causal models, which never produces the same language twice. This novel pretraining strategy forces the model to infer language properties from context, enabling it to predict natural language with high fidelity (0.9-2.4 bits/byte on Wikipedia) and perform complex reasoning tasks on non-linguistic sequences without explicit supervision. The PFLM demonstrates an emergent ability to "learn to learn" languages and other structured domains by generalizing from statistical regularities observed in its synthetic training.
A transformer pretrained on synthetic, never-repeating "languages" learns to infer and predict real-world text and perform complex reasoning tasks, suggesting a fundamental mechanism for language acquisition.
We present the Prior-Fitted Language Model (PFLM), a 300M-parameter byte-level transformer pretrained only on samples from a synthetic non-linguistic prior. Given a prefix of real text, it learns to predict the language in context with frozen weights, having never seen a word of any real language. Every training sequence is generated by a recurrent structural causal model drawn fresh from a distribution over such models. The model never sees the same language twice during training, so the only way to predict the continuation is to infer the language from the prefix. Samples from this prior share the statistical signatures of natural text: Zipfian frequencies, slow entropy-rate convergence, and long-range dependence. On Wikipedia in six languages, bits per byte fall from the uniform eight to between 0.9 and 2.4 at one million bytes of context. Given numerals instead of text, PFLM learns to count, to compare magnitudes, and to add approximately. It predicts deterministic sequences like Rudin-Shapiro or the prime indicator, and it compresses six non-text domains, from source code to speech, below gzip and PPMd. The model has not learned a language. It has learned to learn one.