Qiao Zhao

Forecasting future coding tasks can yield a dataset that is 58.1% relevant to real-world software engineering needs, sidestepping the pitfalls of historical data replay.

LMs don't incrementally track entities through state changes like you'd expect, but instead use a surprisingly fragile "global suppression tag" to handle removals, leading to predictable failure modes.

Turns out, LLMs rely far more on raw code access than documentation when answering repository-level questions, challenging the assumption that documentation is the primary driver of code understanding.