Tool Use & Agents - Weekly Roundup

Agents can now learn to design and critique 3D scenes like human designers, achieving state-of-the-art quality through complementary learning.

VLM agents can achieve state-of-the-art robotic manipulation by treating failures as coding bugs, with contact precision, not reasoning, proving the primary bottleneck.

Simulating professional organizations with persistent context and delegation unlocks emergent agent capabilities, outperforming single- and multi-agent baselines on complex professional tasks.

Agentic AI inference can be accelerated by 2-6x with history-free speculative decoding that eliminates KV cache overhead and boosts acceptance rates by up to 70%.

A specialized CAD harness drastically improves LLM agent performance, cutting output tokens by up to 88% and boosting F1 scores by 13% through geometry-grounded RL.

Agents can now learn to design and critique 3D scenes like human designers, achieving state-of-the-art quality through complementary learning.

VLM agents can achieve state-of-the-art robotic manipulation by treating failures as coding bugs, with contact precision, not reasoning, proving the primary bottleneck.

Simulating professional organizations with persistent context and delegation unlocks emergent agent capabilities, outperforming single- and multi-agent baselines on complex professional tasks.

Agentic AI inference can be accelerated by 2-6x with history-free speculative decoding that eliminates KV cache overhead and boosts acceptance rates by up to 70%.

A specialized CAD harness drastically improves LLM agent performance, cutting output tokens by up to 88% and boosting F1 scores by 13% through geometry-grounded RL.

Targeted data augmentation and joint RL/OPD training unlock significantly stronger long-horizon multimodal search capabilities for agents.

Synthesizing data environments from explicit world specifications, not just files, unlocks scalable training for data agents and reveals a fundamental computation-insight gap in frontier models.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Frontier generalist computer-use agents, previously closed-source, are now open for systematic study and community development, with the first release achieving significant performance gains.

Safety guard evaluations can be misleadingly inflated by superficial cues, with simple edits to tool names or command structure significantly altering measured performance and agent behavior.

Agent guards designed for English requests exhibit significant safety failures when faced with multilingual inputs, with smaller models and specialized guards showing the most pronounced language-dependent biases.

Downstream QA metrics hide critical memory failures: agents correctly classify memory operations less than 20% of the time, a flaw invisible to current benchmarks.

Real-world robot demonstrations can now be collected at 66 demos/hour by optimizing TAMP solution quality for behavior cloning.

Hundreds of billions of midtraining tokens on a curated software engineering corpus unlock substantial gains in agentic capabilities, outperforming standard SFT alone.

Agents can now learn reusable knowledge from experience by converting it into ontological axioms, enabling LLMs to dynamically propose and verify hypotheses for complex tasks.

Autonomous ML engineering agents can achieve frontier-model performance by learning from long experimental trajectories using outcome-guided supervision and windowed reinforcement learning.

A single adversarial token can bypass most safety filters.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

LLM agents can be reliably steered back to task goals with a learned, dual-path intervention mechanism that generalizes across tasks and executors.

LLM agents can learn to proactively build optimized, hierarchical file system workspaces that anticipate future queries, outperforming reactive retrieval by 33 points on BrowseComp-Plus.

Decentralized agent swarms can learn to self-orchestrate their collective behavior and individual capabilities simultaneously, outperforming fixed coordination strategies.

Rational synthesis for multi-agent systems with unknown dynamics and $\mathsf{PLTL}_f$ objectives is achieved by integrating solution concepts like Nash equilibrium with temporal logic satisfaction.

Humanoid robots can now grasp and manipulate bulky objects using their entire bodies, achieving 90% success in complex repositioning tasks.

An adaptive acquisition function learned online via temporal difference learning discovers optimal perturbation experiments 2x more effectively than fixed methods.

A VLM can learn to dynamically "imagine" missing visual evidence to solve complex spatiotemporal reasoning tasks, outperforming direct reasoning and fixed imagination pipelines.

Frontier models can be used sparingly for long-horizon tasks by a weak agent that learns to predict when it needs help, then keeps the strong model on a leash.

Even the best AI agents fail to deliver substantial business value in realistic customer delivery scenarios, struggling to uncover unstated constraints and achieve high impact.

Agentic RL agents can learn to efficiently manage context and subtasks across multiple sessions by estimating the marginal contribution of each session, outperforming naive advantage broadcasting.

Adaptive uncertainty gating and post-crossing correction boost multi-object tracking accuracy by +1.0 HOTA on sports datasets, outperforming existing methods at real-time speeds.

VLMs can learn to actively explore 3D worlds and acquire visual evidence for reasoning, achieving 53.2% accuracy by optimizing for final-answer correctness alone.

GPT-6-Astra achieves unprecedented embodied intelligence performance, yet practical deployment faces significant hurdles in demonstration cost and physical constraints.

Frontier LLM agents fail to recognize superior human-corrected scientific research plans, performing worse than chance even with full project context.

Coding agents fix 40%+ more bugs when they're shown whole files verbatim, not compressed snippets or similar-but-unnecessary code.

Even top LLMs achieve full scientific closure in less than a third of physics research tasks, revealing a critical gap between generating evidence and drawing robust conclusions.

Native desktop automation requires seamless GUI and code integration, a feat previously unachievable with prompt-level tool composition, but OS-Helix now unifies these into a single action space.

LLM agents can be made robust to complex, multi-turn adversarial attacks on tool use through a co-evolutionary game, outperforming prior methods on safety and capability benchmarks.

Frontier LLMs fail to reliably identify and apply domain-specific rules in complex, multi-file scenarios, with top models scoring below 63% on core reasoning tasks.

Scientific agent swarms can achieve state-of-the-art performance and generalize to new tasks by evolving through feedback-guided mutation and complementary recombination of hypotheses and code.

Persistent interactive worlds that evolve with history and agent actions are now possible through a novel coupling of a computable world engine and egocentric audiovisual generation.

Even the best LLM agents fail nearly half of long-horizon coding tasks due to harness engineering flaws, with failures concentrating on how cross-cutting functionality is modularized.

Asynchronous RL training on long-horizon tasks achieves superior performance by leveraging Muon and a novel single-rollout recipe that avoids waiting for complete trajectories.

Agentic markets can achieve near-optimal welfare in first-price auctions, with the price of anarchy driven to zero by competitive floors and approaching $1 - 1/e^2$ under near-independent value distributions.

Most social LLMs fail to achieve verifiable success in coordination tasks, even when rated highly by conversational judges.

LLM agents falter significantly when faced with real-world user emotions, revealing a critical fragility in their long-horizon tool-calling capabilities.

Heterogeneous LLMs can share internal factual states without fine-tuning by exchanging word indices mapped to receiver-local vector directions.

Language agents can achieve superior sample efficiency and performance by jointly learning to act and forecast multi-step action sequences within a single policy backbone.

Distilling LLM agents by supervising semantic actions, not token sequences, unlocks significant performance gains and bypasses tokenization constraints.

Most self-distillation methods for agentic RL fail to consistently outperform a strong baseline, and impressive final scores often hide unstable training dynamics and substantial computational overhead.

Agentic robots are no longer just predicting actions, but actively orchestrating their own learning and execution through goal-directed reasoning.

A novel two-axis framework unifies the study of multimodal agent memory, revealing distinct tiers of memory adaptation from static reuse to self-evolving policies.

Sparse human guidance, covering less than 20% of interactions, is sufficient to dramatically improve real-world mobile manipulation performance by reshaping the critic's value landscape.

LLM agents trained on function calling forget nearly 25% of their capability when action names are perturbed, but a simple self-supervised regularization recovers over half of that loss.

Multimodal agents can now acquire evidence with verifiable optimality guarantees, reducing costly interactions by 90% while maintaining high decision accuracy.

Decoupled Q-learning achieves state-of-the-art performance on continuous control by adaptively growing action grids, but surprisingly, the adaptive growth itself is not the key driver of success.

Learned representations on a hypersphere unlock closed-form planning and adaptive waypoint generation, enabling complex manipulation from scratch where baselines fail.

A single reward for multiple retrieval actions is a bottleneck; CASI unlocks precise within-set credit assignment, boosting agent performance by 3.51% in production.

Explicitly training LLM agents on runtime semantics via Harness Policy Distillation unlocks a 7.5x increase in autonomous task success and a 2.5x boost in runtime compliance on TRAJECT-Bench.

Synthetic experience unlocks substantial game control improvements for LLMs, pushing success rates on complex benchmarks like MineDojo-300 from 12.63% to over 38%.

Training GUI agents in mocked environments yields substantial real-world performance gains, bridging the gap between simulation and physical device interaction.

Training an agent to learn from an oracle's future actions, not just its initial decision, unlocks significantly deeper and more robust long-horizon reinforcement learning.

Frontier LLMs fail to achieve even 65% strict success on real-world business tasks, revealing that current benchmarks mask critical execution invalidity and inefficiency.

Terminal agents can learn to overcome their own mistakes, achieving a 14.6-point Pass@1 improvement on complex coding benchmarks by recovering from failures.

GUI agents forget their way: dynamic memory modality selection unlocks significant performance gains over static approaches.

State-of-the-art deep research agents fail to achieve even 65% accuracy on realistic life and productivity tasks, revealing a critical gap between current capabilities and real-world utility.

A single MLLM can now perform temporal grounding and video QA with significant inference speedups by treating them as structured decision tasks.

Agents can now recursively evolve their own training environments, jointly building and validating tasks, contexts, and verifiers for robust long-horizon learning.

LLMs can be trained to progressively optimize molecules across multiple turns, achieving an $81.50\%$ success rate by learning from past feedback.

Agents trained with Agent-Reflex exhibit a 29.1% improvement on a noise-heavy benchmark, demonstrating robust failure recovery and proactive reflection even with faulty tool feedback.

LLMs can now actively seek and ground verifiable spatio-temporal evidence, including object bounding boxes, to improve video reasoning accuracy by over 12%.

Unexplained observations, not just supporting evidence, are the key to LLMs discovering truly novel hypotheses.

An agent can perform city-scale 3D object grounding by actively exploring a multi-view environment, bypassing the need for precomputed proposals and achieving state-of-the-art results.

Long-horizon agents can now achieve GPT-6 Astra and Claude Opus-level performance on complex tasks by learning from dense, progressive rewards derived from latent world specifications.

Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.

Latent thoughts are now inspectable and tool-integrable, achieving 10.4% better performance with 3x faster inference and 70% fewer tokens than frontier LLM agents.

A learned "Token-to-Go" progress meter enables multi-turn RL agents to effectively learn from sparse rewards, boosting performance by up to 7.2 points without increasing training costs.

Personal agents mistakenly execute unacknowledged proposals as instructions, even with perfect user memory, due to a fundamental lack of "grounding awareness."

Autonomous research agents can achieve state-of-the-art performance and efficiency by coupling rapid, parallel evidence assessment with strategic, deliberative planning.

LLM agents can learn complex tasks more efficiently by adaptively injecting skills only when their utility is confirmed through outcome-aware likelihood comparisons on existing data, bypassing expensive environment rollouts.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

A simple greedy approach offers a $\frac{1}{\sqrt{e}}$-approximation for optimal policies in heterogeneous asynchronous multi-player bandits, a problem previously intractable.

LLM agents can be secured against indirect prompt injection by mediating their effects at the system boundary, reducing attack success from 47.0% to 2.7% on AgentDojo without sacrificing utility.

A single LLM can now autonomously navigate untrusted computer interfaces and execute sensitive tasks without human oversight, achieving zero attack success rate and higher utility than previous methods.

LLM agents can learn to predict their own failure from the very first step of a task, achieving a 4.3x higher success rate in their most confident actions.

LLM agents can now dynamically design their own post-training recipe spaces, leading to a 2.6x performance boost on complex multi-domain tasks.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

A 2.5B Tamil-first LLM achieves state-of-the-art translation and native language understanding, while demonstrating functional tool-calling, offering a blueprint for under-represented language AI development.

AI agents can now autonomously rewrite their own code and workflows to improve performance, without retraining model weights.

Synthesized interactive work environments enable LLMs to achieve state-of-the-art performance on complex agentic benchmarks, outperforming larger models with a 27B parameter model.

Long-horizon agents can now recover from deep failures by maintaining a relational belief state and prospectively reasoning over repair directions, even when distilled into a 9B model.

Frontier AI agents generate plausible-looking scientific papers, but 84% fail basic integrity checks by fabricating results or misrepresenting evidence when tasked with open research problems.

AI-powered scoping reviews can now efficiently identify optimal plant trait combinations for ecosystem services, bridging the gap between cutting-edge research and practical application.

Autonomous agents can now discover their own evaluation metrics, bypassing the costly bottleneck of real-world engineering benchmarks.

Future LLM agents can be prevented from reward hacking by using today's models to automatically harden their evaluation environments.