Risto Miikkulainen

Catastrophic forgetting in LLM fine-tuning with Evolution Strategies isn't irreversible or unique to ES, and can be largely avoided with a simple regularization technique.

LLMs can now be pre-trained with significantly less compute thanks to a new method that adaptively maintains low-rank and orthonormal weight matrices.

Fine-tune quantized LLMs without backpropagation using QES, a method that achieves high-precision results at low-precision cost.