Taiqiang Wu

Attention Sink, where Transformers fixate on seemingly irrelevant tokens, is more than just a quirk – it's a fundamental challenge impacting training, inference, and even causing hallucinations, demanding a systematic approach to understanding and mitigating its effects.

Autonomous driving's next leap hinges on reasoning, not just perception, but current LLM-based approaches are too slow for real-time control.

Training LLMs for efficient reasoning is best achieved by using easier prompts to ensure a dense positive reward signal, preventing undesirable length collapse.