Code Generation & Program Synthesis

Coding agents routinely fail to build agents that truly meet user requirements, with the gap between demonstration and delivery being the norm.

Human GUI interactions can be precisely captured and translated into reusable agent skills, dramatically boosting performance on unseen software and workflows.

Spatial coding agents can be distilled into standalone MLLMs, achieving superior spatial reasoning and generalization without external tools.

LEGO-building AI agents can assemble physically valid structures from text prompts, but their designs lag significantly behind human creativity and complexity.

Robots can achieve 70% success on complex bimanual tasks using only code for decision-making, bypassing runtime vision-language models entirely.

Weisfeiler-Lehman graph kernels unlock superior structural features for algorithm selection, outperforming traditional methods on challenging constraint satisfaction problems.

LLMs can now generate executable feature programs from long text, enabling efficient structured data prediction without real-time LLM inference.

LLM test generation quality is overestimated by 30% when evaluated against single references, with a new metric revealing widespread missed behaviors and faulty assertions.

Most AI agent skills copied on GitHub originate from a few hidden hubs, rendering security patches ineffective for the vast majority of users.

Coding agent benchmarks can now dynamically scale difficulty and avoid data leakage by having agents evaluate each other, with the latest models generating tasks that defeat older solvers 10x more often.

Synthesized software environments unlock a 50% boost in CUA performance on unseen benchmarks, proving scalable supervision is key.

Automated kernel discovery now yields reusable inductive biases that generalize beyond training data, outperforming meta-learned deep kernels and even human-refined versions on unseen tasks.

Small Language Models can parse ML pipeline structures, but their insights into data science practices diverge significantly from other classifiers, highlighting the critical impact of taxonomy definition.

Current LLMs struggle to translate visual scenes into executable geometric functions, failing even basic trajectory generation tasks.

Key contribution not extracted.

Real-world coding agent usage reveals that task flow, not just task completion, dramatically impacts agent cost and effectiveness, suggesting current benchmarks are misaligned with user behavior.

Broader tokenizer alignment in on-policy distillation can actively harm performance by introducing noisy supervision signals.

Sparse MoE models match dense models' code repair correctness with drastically fewer active parameters, challenging the sole reliance on total parameter count for evaluating code generation efficiency.

Automated code generation from 3DGS papers not

LLMs can now autonomously debug complex chip designs by translating natural language queries into SQL against massive simulation dumps.

Reasoning in base LLMs can be unlocked by simply prepending specific tokens, matching RL-tuned performance on math and coding.

GPU kernel optimization agents can regress by up to 19% when transferring distilled knowledge across DSLs, revealing that source attempt quality, not just success, dictates transfer efficacy.

Achieve state-of-the-art LLM-guided program evolution for under $2, drastically cutting costs for complex optimization tasks.

LLM-driven heuristic design now learns from noisy system logs to uncover hidden system dynamics, outperforming prior methods in real-world resource assignment.

Even the largest open-weight LLMs fail to generate accurate cybersecurity CLI commands over 58% of the time, but fine-tuning with verifiable rewards on KaliBench closes the gap with massive MoE models.

Mutation-enhanced skill evolution unlocks significant improvements in Lean theorem proving by intelligently sampling mathematical concepts when proof generation stalls.

Verifier-guided RL unlocks a 34% accuracy boost in mathematical reasoning by directly penalizing incorrect intermediate steps, not just final answers.

Coding agents can achieve over 4x throughput gains with a novel retrieval-based speculative decoding framework that intelligently sources and formats reusable text and adapts draft lengths.

Even the best AI agents fail to generate engineering-grade Simulink models, scoring below 43% on a new benchmark that rigorously tests functional correctness and robustness.

A listwise agent inspecting top candidates holistically boosts patch retrieval accuracy by over 27% compared to treating them independently.

AutoResearch systems can achieve up to 28% fewer errors by making competing behaviors visible and actively steering edits to resolve their inherent conflicts.

Lightweight sentinels can learn to proactively intervene in coding agent actions, boosting task success by up to 14% by preventing costly error propagation.

Autonomous agents can recursively self-improve LLM-generated games, achieving superhuman performance and drastically cutting generation costs by internalizing development experience.

Frozen tabular foundation models can dominate competitive ML benchmarks without parameter updates if an LLM agent is tasked strictly with evolving the semantic data pipeline around them.

Self-improving coding agents do not require downstream reward signals to evolve: bootstrapping purely from historical self-modification traces matches the performance of benchmark-guided search at a fraction of the cost.

Robotic agents can now learn complex manipulation strategies from just a few demonstrations by treating them as evidence to be analyzed, not just training data.

Modifying executable agent harnesses rather than model weights allows test-time adaptation to generalize across unseen reasoning tasks without a single gradient step.

Aggregate benchmark scores actively mislead in quantum computing, failing to predict individual model competence across more than a third of tasks like circuit compilation and quantum error correction.

Non-CUDA hardware no longer requires costly post-training pipelines: conditioning memory rewards strictly on adopted code enables zero-shot LLM agents to synthesize NPU kernels that outperform vendor libraries by up to 44x.

Instead of spending compute fitting bespoke generative models for every dataset, a single feedforward pass can now compile empirical samples directly into inspectable, executable generative programs.

Frontier coding agents can now write 100% functionally correct GPU physics simulations, yet they match human-expert performance on just 22% of tasks—revealing a stark capability gap between generating working code and producing performant systems.

Standard pass rates hide severe technical debt: coding agents duplicate logic in over half of multi-turn task chains by turn 5 while progressively abandoning repository exploration.

Across fifteen read vectors, standard containers and sandboxes completely fail to distinguish an autonomous coding agent from the compiler it executes, leaving proprietary dependencies exposed to exfiltration.

Human-engineered black-box optimizers have been eclipsed by a recursive dual-agent system that autonomously evolves both the target optimizer codebase and its own meta-design logic from scratch.

Constrained decoding no longer has to stop at syntax: embedding type and scope checks directly into Earley parser descent prunes dead-end code mid-stream to deliver up to a 15-point gain in program correctness.

Coding agents are failing to demonstrate robust engineering skills in robotics, with RLE-Bench revealing significant gaps in multimodal reasoning and artifact integration beyond simple control.

DataMagic achieves over 95% success rate and an 83% quality improvement in data video generation by declaratively orchestrating multi-agent scene creation, surpassing even state-of-the-art LLMs.

Coding agents can now achieve up to 15% higher task completion rates by treating feedback as a persistent note that's followed until resolved, not just a one-off evaluation.