Kaifeng Lyu

Achieving near state-of-the-art performance for under $7K opens the door for cost-effective language model training accessible to the broader research community.

Off-policy distillation fails in multi-task settings, but a two-phase approach combining it with on-policy refinement can achieve single-task expert performance across multiple tasks.

Safety Reflection Pretraining cuts down the success of inference attacks by embedding self-monitoring directly into LLMs during pretraining.

Hyperball achieves a remarkable 20-30% speedup in language model pretraining, even as model sizes grow, challenging the limits of traditional optimizers.

LLMs miss over 50% of errors in human-written code, but with test-time scaling, they can identify issues in more than 90% of cases—if you can afford the compute.

MLLMs are surprisingly robust to catastrophic forgetting during fine-tuning, needing only simple regularization or data-hybrid training to maintain performance.

Resource-constrained LLM pretraining gets a boost with PCMind-2.1-Kaiyuan-2B's open-source release and novel techniques for data mixing, repetition, and curriculum learning.