Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces distributional hypernetworks that learn to predict a distribution over LoRA weight updates conditioned solely on the input query, enabling test-time adaptation via sampling weight updates rather than token sequences. The proposed end-to-end training with differentiable Monte Carlo approximation allows for generating multiple adapted models for a single query, outperforming deterministic hypernetworks and token-sampling baselines. The learned distributions not only improve performance with increased sampling but also demonstrate transferability of adaptation strategies across queries, suggesting a novel approach to both adaptation and test-time scaling.
Sampling weight updates, not just tokens, unlocks significant test-time adaptation gains for LLMs.
Hypernetworks have recently shown success in dynamically adapting the parameters of Large Language Models (LLMs) at runtime based on signals such as task descriptions or additional demostrations. Here we ask: how much adaptation signal can be obtained using only the input query to an LLM?. To answer this, we study query-conditioned Hypernetworks for LoRA estimation. Further, we introduce distributional Hypernetworks, able to produce not only point estimates of parameter adaptors, but also a distribution over possible LoRAs. For this we propose a simple end-to-end loss using a differentiable Monte Carlo approximation and explore multiple distribution parametrizations including regression and convex combination variants. Results show that even using the mean of the learned distribution can outperform deterministic hypernetworks. Crucially, the learned distribution enables a different form of test-time scaling: instead of spending additional compute only by sampling more token sequences from a fixed model, we sample weight updates, yielding multiple adapted models for the same query. Performance improves as more weight samples are considered and remains stronger than corresponding token-sampling adaptation baselines. Finally, we find that generated updates can transfer across queries, suggesting that the hypernetwork learns reusable structure in how the model should adapt. Together, these results show that query-conditioned distributions over weight updates can support both adaptation and test-time scaling.