Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to long-context fine-tuning by integrating Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered key-value (KV) storage, allowing for efficient training on sequences of up to 16,384 tokens with limited VRAM. By detaching older KV data into RAM or NVMe, the method enables the active segment to remain differentiable in VRAM, significantly enhancing the model's capacity without exceeding memory limits. The results demonstrate that HGA achieves comparable performance to dense training while being faster and more memory-efficient, suggesting a promising direction for scaling model context in resource-constrained environments.
HGA enables fine-tuning on sequences of 16,384 tokens with just 15.28 GB of VRAM, outperforming traditional methods constrained by memory limits.
Parameter-efficient fine-tuning reduces model and optimizer memory, but dense attention still makes long training sequences expensive. We combine Hierarchical Global Attention (HGA) with segment-wise backpropagation and tiered KV storage. Only the active segment remains differentiable in VRAM; older KV is detached into RAM or NVMe, and HGA loads a bounded set of exact historical tokens for each query block. On Qwen3-8B with 4-bit QLoRA and PG19, dense training on a 16 GB Quadro RTX 5000 fits 2,048 tokens but fails at 4,096, whereas HGA reaches 16,384 tokens with 15.28 GB peak VRAM. Under evaluation the same adapter runs through 131,072 tokens on this card; VRAM is not constant but grows gently with the resident chunk summaries, so RAM and NVMe capacity set the practical limit beyond these lengths. At the shared 2K training length, HGA-trained and dense-trained adapters obtain 2.7405 and 2.7383 nat under the same dense-attention readout, while the stock model obtains 2.9541. At this boundary HGA training is already marginally faster (217.75 vs. 207.02 tokens/s), and the HGA-to-dense throughput ratio improves from 1K to 2K; because HGA keeps the attended historical set per token approximately constant while dense work per token grows, we expect this lead to widen as context grows. Dense attention is used for the main quality and retrieval comparisons so that they measure the learned weights and remain compatible with standard generation frameworks. HGA can also be used for retrieval and generation; an optimized production-grade serving implementation is under development.
