Tool Use & Agents - Weekly Roundup

A specialized CAD harness drastically improves LLM agent performance, cutting output tokens by up to 88% and boosting F1 scores by 13% through geometry-grounded RL.

Targeted data augmentation and joint RL/OPD training unlock significantly stronger long-horizon multimodal search capabilities for agents.

Synthesizing data environments from explicit world specifications, not just files, unlocks scalable training for data agents and reveals a fundamental computation-insight gap in frontier models.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Frontier generalist computer-use agents, previously closed-source, are now open for systematic study and community development, with the first release achieving significant performance gains.

A specialized CAD harness drastically improves LLM agent performance, cutting output tokens by up to 88% and boosting F1 scores by 13% through geometry-grounded RL.

Targeted data augmentation and joint RL/OPD training unlock significantly stronger long-horizon multimodal search capabilities for agents.

Synthesizing data environments from explicit world specifications, not just files, unlocks scalable training for data agents and reveals a fundamental computation-insight gap in frontier models.

A strong LLM can review grounded drafts from a weak model, retaining most task performance while halving latency and memory traffic.

Frontier generalist computer-use agents, previously closed-source, are now open for systematic study and community development, with the first release achieving significant performance gains.

Safety guard evaluations can be misleadingly inflated by superficial cues, with simple edits to tool names or command structure significantly altering measured performance and agent behavior.

Agent guards designed for English requests exhibit significant safety failures when faced with multilingual inputs, with smaller models and specialized guards showing the most pronounced language-dependent biases.

Downstream QA metrics hide critical memory failures: agents correctly classify memory operations less than 20% of the time, a flaw invisible to current benchmarks.

Real-world robot demonstrations can now be collected at 66 demos/hour by optimizing TAMP solution quality for behavior cloning.

Hundreds of billions of midtraining tokens on a curated software engineering corpus unlock substantial gains in agentic capabilities, outperforming standard SFT alone.

Agents can now learn reusable knowledge from experience by converting it into ontological axioms, enabling LLMs to dynamically propose and verify hypotheses for complex tasks.

Autonomous ML engineering agents can achieve frontier-model performance by learning from long experimental trajectories using outcome-guided supervision and windowed reinforcement learning.

A single adversarial token can bypass most safety filters.

Sharing LLM internal states is efficient, but only if you augment the KV cache and replace activations, not swap them.

LLM agents can be reliably steered back to task goals with a learned, dual-path intervention mechanism that generalizes across tasks and executors.

LLM agents can learn to proactively build optimized, hierarchical file system workspaces that anticipate future queries, outperforming reactive retrieval by 33 points on BrowseComp-Plus.

Decentralized agent swarms can learn to self-orchestrate their collective behavior and individual capabilities simultaneously, outperforming fixed coordination strategies.

Rational synthesis for multi-agent systems with unknown dynamics and $\mathsf{PLTL}_f$ objectives is achieved by integrating solution concepts like Nash equilibrium with temporal logic satisfaction.

Native desktop agents can finally bridge the gap between visual GUI interaction and programmatic code execution by learning a unified action space, achieving state-of-the-art performance on complex real-world tasks.

Humanoid robots can now grasp and manipulate bulky objects using their entire bodies, achieving 90% success in complex repositioning tasks.

An adaptive acquisition function learned online via temporal difference learning discovers optimal perturbation experiments 2x more effectively than fixed methods.

A VLM can learn to dynamically "imagine" missing visual evidence to solve complex spatiotemporal reasoning tasks, outperforming direct reasoning and fixed imagination pipelines.

Frontier models can be used sparingly for long-horizon tasks by a weak agent that learns to predict when it needs help, then keeps the strong model on a leash.

Even the best AI agents fail to deliver substantial business value in realistic customer delivery scenarios, struggling to uncover unstated constraints and achieve high impact.

Agentic RL agents can learn to efficiently manage context and subtasks across multiple sessions by estimating the marginal contribution of each session, outperforming naive advantage broadcasting.

LLM agents can be made robust to complex, multi-turn adversarial attacks on tool use through a co-evolutionary game, outperforming prior methods on safety and capability benchmarks.

Frontier LLMs fail to reliably identify and apply domain-specific rules in complex, multi-file scenarios, with top models scoring below 63% on core reasoning tasks.

Scientific agent swarms can achieve state-of-the-art performance and generalize to new tasks by evolving through feedback-guided mutation and complementary recombination of hypotheses and code.

Persistent interactive worlds that evolve with history and agent actions are now possible through a novel coupling of a computable world engine and egocentric audiovisual generation.

Even the best LLM agents fail nearly half of long-horizon coding tasks due to harness engineering flaws, with failures concentrating on how cross-cutting functionality is modularized.

Asynchronous RL training on long-horizon tasks achieves superior performance by leveraging Muon and a novel single-rollout recipe that avoids waiting for complete trajectories.

Agentic markets can achieve near-optimal welfare in first-price auctions, with the price of anarchy driven to zero by competitive floors and approaching $1 - 1/e^2$ under near-independent value distributions.

Most social LLMs fail to achieve verifiable success in coordination tasks, even when rated highly by conversational judges.

LLM agents falter significantly when faced with real-world user emotions, revealing a critical fragility in their long-horizon tool-calling capabilities.

Heterogeneous LLMs can share internal factual states without fine-tuning by exchanging word indices mapped to receiver-local vector directions.

Language agents can achieve superior sample efficiency and performance by jointly learning to act and forecast multi-step action sequences within a single policy backbone.

Distilling LLM agents by supervising semantic actions, not token sequences, unlocks significant performance gains and bypasses tokenization constraints.

Most self-distillation methods for agentic RL fail to consistently outperform a strong baseline, and impressive final scores often hide unstable training dynamics and substantial computational overhead.

Agentic robots are no longer just predicting actions, but actively orchestrating their own learning and execution through goal-directed reasoning.

A novel two-axis framework unifies the study of multimodal agent memory, revealing distinct tiers of memory adaptation from static reuse to self-evolving policies.

Sparse human guidance, covering less than 20% of interactions, is sufficient to dramatically improve real-world mobile manipulation performance by reshaping the critic's value landscape.

LLM agents trained on function calling forget nearly 25% of their capability when action names are perturbed, but a simple self-supervised regularization recovers over half of that loss.

Multimodal agents can now acquire evidence with verifiable optimality guarantees, reducing costly interactions by 90% while maintaining high decision accuracy.

Decoupled Q-learning achieves state-of-the-art performance on continuous control by adaptively growing action grids, but surprisingly, the adaptive growth itself is not the key driver of success.

Learned representations on a hypersphere unlock closed-form planning and adaptive waypoint generation, enabling complex manipulation from scratch where baselines fail.

A single reward for multiple retrieval actions is a bottleneck; CASI unlocks precise within-set credit assignment, boosting agent performance by 3.51% in production.

Explicitly training LLM agents on runtime semantics via Harness Policy Distillation unlocks a 7.5x increase in autonomous task success and a 2.5x boost in runtime compliance on TRAJECT-Bench.

Synthetic experience unlocks substantial game control improvements for LLMs, pushing success rates on complex benchmarks like MineDojo-300 from 12.63% to over 38%.

Training GUI agents in mocked environments yields substantial real-world performance gains, bridging the gap between simulation and physical device interaction.

Training an agent to learn from an oracle's future actions, not just its initial decision, unlocks significantly deeper and more robust long-horizon reinforcement learning.

Terminal agents can learn to overcome their own mistakes, achieving a 14.6-point Pass@1 improvement on complex coding benchmarks by recovering from failures.

GUI agents forget their way: dynamic memory modality selection unlocks significant performance gains over static approaches.

State-of-the-art deep research agents fail to achieve even 65% accuracy on realistic life and productivity tasks, revealing a critical gap between current capabilities and real-world utility.

A single MLLM can now perform temporal grounding and video QA with significant inference speedups by treating them as structured decision tasks.

Agents can now recursively evolve their own training environments, jointly building and validating tasks, contexts, and verifiers for robust long-horizon learning.

LLMs can now learn to progressively optimize molecules across multiple turns by explicitly conditioning on past feedback, achieving substantial gains in similarity-constrained success rates.

Agents trained with Agent-Reflex exhibit a 29.1% improvement on a noise-heavy benchmark, demonstrating robust failure recovery and proactive reflection even with faulty tool feedback.

LLMs can now actively seek and ground verifiable spatio-temporal evidence, including object bounding boxes, to improve video reasoning accuracy by over 12%.

Unexplained observations, not just supporting evidence, are the key to LLMs discovering truly novel hypotheses.

An agent can perform city-scale 3D object grounding by actively exploring a multi-view environment, bypassing the need for precomputed proposals and achieving state-of-the-art results.

Long-horizon agents can now achieve GPT-6 Astra and Claude Opus-level performance on complex tasks by learning from dense, progressive rewards derived from latent world specifications.

Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.

Latent thoughts are now inspectable and tool-integrable, achieving 10.4% better performance with 3x faster inference and 70% fewer tokens than frontier LLM agents.

A learned "Token-to-Go" progress meter enables multi-turn RL agents to effectively learn from sparse rewards, boosting performance by up to 7.2 points without increasing training costs.

Personal agents mistakenly execute unacknowledged proposals as instructions, even with perfect user memory, due to a fundamental lack of "grounding awareness."

Autonomous research agents can achieve state-of-the-art performance and efficiency by coupling rapid, parallel evidence assessment with strategic, deliberative planning.

LLM agents can learn complex tasks more efficiently by adaptively injecting skills only when their utility is confirmed through outcome-aware likelihood comparisons on existing data, bypassing expensive environment rollouts.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

A simple greedy approach offers a $\frac{1}{\sqrt{e}}$-approximation for optimal policies in heterogeneous asynchronous multi-player bandits, a problem previously intractable.

LLM agents can be secured against indirect prompt injection by mediating their effects at the system boundary, reducing attack success from 47.0% to 2.7% on AgentDojo without sacrificing utility.

A single LLM can now autonomously navigate untrusted computer interfaces and execute sensitive tasks without human oversight, achieving zero attack success rate and higher utility than previous methods.

LLM agents can learn to predict their own failure from the very first step of a task, achieving a 4.3x higher success rate in their most confident actions.

LLM agents can now dynamically design their own post-training recipe spaces, leading to a 2.6x performance boost on complex multi-domain tasks.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

A 2.5B Tamil-first LLM achieves state-of-the-art translation and native language understanding, while demonstrating functional tool-calling, offering a blueprint for under-represented language AI development.

AI agents can now autonomously rewrite their own code and workflows to improve performance, without retraining model weights.

Synthesized interactive work environments enable LLMs to achieve state-of-the-art performance on complex agentic benchmarks, outperforming larger models with a 27B parameter model.

AI-powered scoping reviews can uncover novel plant trait combinations for ecosystem services, outperforming purely manual efforts.

Future LLM agents can be prevented from reward hacking by using today's models to automatically harden their evaluation environments.