Search papers, labs, and topics across Lattice.
This paper introduces a framework for deriving new language models by applying deterministic string-to-string transformations, specifically finite-state transducers (FSTs), to existing language models. They develop algorithms for marginalizing over source strings and conditioning on transformed outputs, enabling probability propagation through the FST without retraining the original LM. Experiments across token-to-byte, token-to-word, and DNA-to-amino acid transformations demonstrate the framework's ability to adapt pretrained LMs to application-specific output requirements at inference time.
Repurpose existing language models for entirely new output formats like bytes, words, or even DNA sequences, all without retraining, by wrapping them in a finite-state transducer.
Modern language models define distributions over strings, but downstream tasks often require different output formats. For instance, a model that generates byte-pair strings does not directly produce word-level predictions, and a DNA model does not directly produce amino-acid sequences. In such cases, a deterministic string-to-string transformation can convert the model's output to the desired form. This is a familiar pattern in probability theory: applying a function $f$ to a random variable $X\sim p$ yields a transformed random variable $f(X)$ with an induced distribution. While such transformations are occasionally used in language modeling, prior work does not treat them as yielding new, fully functional language models. We formalize this perspective and introduce a general framework for language models derived from deterministic string-to-string transformations. We focus on transformations representable as finite-state transducers -- a commonly used state-machine abstraction for efficient string-to-string mappings. We develop algorithms that compose a language model with an FST to *marginalize* over source strings mapping to a given target, propagating probabilities through the transducer without altering model parameters and enabling *conditioning* on transformed outputs. We present an exact algorithm, an efficient approximation, and a theoretical analysis. We conduct experiments in three domains: converting language models from tokens to bytes, from tokens to words, and from DNA to amino acids. These experiments demonstrate inference-time adaptation of pretrained language models to match application-specific output requirements.
