Martin Marek

It is shown that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step, and an additive score centering term is derived that stabilizes RL under TIM by canceling drift.

Language models can leverage their own generative abilities to nearly eliminate forgetting during finetuning, challenging the need for stored exemplars.