Search papers, labs, and topics across Lattice.
The paper introduces HyLo, a method for upcycling pretrained Transformer LLMs into hybrid architectures that combine efficient Transformer components with linear sequence modeling blocks. HyLo leverages architectural adaptation, Multi-Head Latent Attention (MLA), linear blocks (Mamba2 or Gated DeltaNet), staged long-context training, and teacher-guided distillation to extend usable context length by up to 32x and reduce KV-cache memory by over 90%. Experiments on 1B- and 3B-scale models demonstrate that HyLo achieves strong short- and long-context performance, outperforming state-of-the-art upcycled hybrid baselines and even surpassing JetNemotron (trained on 40x more tokens) on several benchmarks.
Forget training from scratch: HyLo lets you breathe new (long-context) life into your existing Transformer LLMs, achieving 32x context extension and 90% KV-cache reduction.
Hybrid sequence models that combine efficient Transformer components with linear sequence modeling blocks are a promising alternative to pure Transformers, but most are still pretrained from scratch and therefore fail to reuse existing Transformer checkpoints. We study upcycling as a practical path to convert pretrained Transformer LLMs into hybrid architectures while preserving short-context quality and improving long-context capability. We call our solution \emph{HyLo} (HYbrid LOng-context): a long-context upcycling recipe that combines architectural adaptation with efficient Transformer blocks, Multi-Head Latent Attention (MLA), and linear blocks (Mamba2 or Gated DeltaNet), together with staged long-context training and teacher-guided distillation for stable optimization. HyLo extends usable context length by up to $32\times$ through efficient post-training and reduces KV-cache memory by more than $90\%$, enabling up to 2M-token prefill and decoding in our \texttt{vLLM} inference stack, while comparable Llama baselines run out of memory beyond 64K context. Across 1B- and 3B-scale settings (Llama- and Qwen-based variants), HyLo delivers consistently strong short- and long-context performance and significantly outperforms state-of-the-art upcycled hybrid baselines on long-context evaluations such as RULER. Notably, at similar scale, HyLo-Qwen-1.7B trained on only 10B tokens significantly outperforms JetNemotron (trained on 400B tokens) on GSM8K, Lm-Harness common sense reasoning and RULER-64K.
