Inderjit S Dhillon

Restricting LLM fine-tuning to the bottom singular subspace nearly eliminates catastrophic forgetting, even when protecting 99% of weights, while maintaining strong target-task performance.

APEX reveals that optimizing data alongside prompts can boost LLM performance by over 11% while significantly reducing wasted compute resources.