Woosung Koh

Small models can now learn new skills without forgetting old ones, thanks to a novel distillation technique that leverages a larger model to guide the learning process.

Language models can now self-select relevant context, slashing attention costs by over 50% while maintaining performance.