Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
LARC introduces a low-rank adaptive residual connection ($h+BAh$) to enable learning in frozen LLMs by maintaining a compact, task-specific numerical state. This method utilizes a slow state ($\rho$) for learning initial factors and a private fast state ($\Phi$) for feedback-driven adaptation, allowing efficient fine-tuning without altering the base model weights. Experiments on MiniCPM5-1B-SFT demonstrate significant reductions in query execution error and improved performance on CI job prediction compared to static baselines.
A low-rank residual connection allows frozen LLMs to learn from feedback with minimal parameters, achieving substantial error reduction in downstream tasks.
Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map $h+BAh$ adds a low-rank correction to a hidden representation. A slow state $ρ$ learns starting factors across tasks; a private fast state $Φ$ copies them, changes with feedback, and resets to the trained initialization. This report specifies an input-side realization of the numerical policy carrier in Memory-Mediated Learning Architecture and examines its factor-space dynamics and learning lifetime. We study a rank-4 input residual with 12,288 trainable parameters on a frozen MiniCPM5-1B-SFT substrate. In a four-candidate program-selection task, two feedback-gradient steps reduce expected query execution error by 24.65 and 36.65 percentage points relative to resetting to the respective trained static and post-adaptation initializations. These development results cover 16 parameter groups and three paired training seeds. A direct support-loss selection rule is much more accurate, reaching 0.78125% error. In a repository-balanced chronological replay of public continuous-integration jobs, retaining online updates raises half-Brier loss from 0.1274 to 0.1808. A fixed follow-up intervention records same-batch non-descent and inconsistent future benefit from shrinking updates. Together, the algebra and measurements distinguish residual capacity, adaptation relative to a starting point, and usefulness on later decisions.