Shun Kiyono

Ditch the learning rate decay: LLMs pre-trained with a stable learning rate after warmup show superior adaptability and downstream performance after supervised fine-tuning compared to those trained with decay-based schedules.