Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Telescopic Language Models (TLMs) replace discrete early-exit training by supervising standard Transformers with stochastic layer-prefix truncation paired with an anchored full-capacity pass. Unlike existing Matryoshka-style nested suites that catastrophically fail at intermediate non-supervised depths, TLM establishes a continuous capacity spectrum where every layer depth functions as an effective language model. On a 200M-parameter benchmark trained on 20B tokens, a single TLM run yields valid models across all 20 depth prefixes, cutting the quality-budget area under the curve by 43–44% at 12% lower GPU training cost without degrading full-capacity performance.
Standard early-exit architectures collapse completely outside their pre-selected exits, but stochastic prefix supervision transforms a normal Transformer into a valid, deployable language model at literally every intermediate layer depth.
One deployed language model must often serve many compute budgets, yet serving each budget still means a separate training or compression run per point. We train a Telescopic Language Model (TLM) to be that continuum: a nested-capacity Transformer supervised by stochastic prefix supervision with a full anchor. At every step, one randomly truncated prefix of the capacity axis is trained against the full next-token target, alongside one full-capacity pass, so the trained artifact is a valid language model at every depth. Two forward-backward passes per step, no architectural change, nothing extra at inference. Fixed-exit suites such as Matryoshka Language Model Suites (MLMS) occupy one point in this design space, and the point has a cost: supervising only a few fixed exits leaves the nested model at chance level everywhere else (perplexity 10^2-10^5 in our baselines). On a 200M proxy suite (20B FineWeb-Edu tokens, identical data stream for all methods), a single TLM run is a valid language model at every one of its twenty layer prefixes, in perplexity and on perplexity-sensitive downstream tasks, reducing the area under the quality-budget curve by 43-44% relative to the fixed-exit suites while matching them at full capacity, at ~12% lower GPU cost per run. The prefix sampling density is a dial: concentrating it on a few depths recovers fixed-exit quality there at the price of the continuum, so the operating points become a training-time choice rather than an architectural one. These results indicate that the training objective, not the nesting itself, is what makes a model elastic.