Tool Use & Agents - Weekly Roundup

Mobile agents can now achieve near-frontier performance by learning to seamlessly blend GUI interactions with app-native tool calls, unlocking complex, long-horizon tasks.

Terminal agents can learn to overcome their own mistakes, achieving a 14.6-point Pass@1 improvement on complex coding benchmarks by recovering from failures.

GUI agents forget their way: dynamic memory modality selection unlocks significant performance gains over static approaches.

State-of-the-art deep research agents fail to achieve even 65% accuracy on realistic life and productivity tasks, revealing a critical gap between current capabilities and real-world utility.

A single MLLM can now perform temporal grounding and video QA with significant inference speedups by treating them as structured decision tasks.

Terminal agents can learn to overcome their own mistakes, achieving a 14.6-point Pass@1 improvement on complex coding benchmarks by recovering from failures.

GUI agents forget their way: dynamic memory modality selection unlocks significant performance gains over static approaches.

State-of-the-art deep research agents fail to achieve even 65% accuracy on realistic life and productivity tasks, revealing a critical gap between current capabilities and real-world utility.

A single MLLM can now perform temporal grounding and video QA with significant inference speedups by treating them as structured decision tasks.

Agents can now recursively evolve their own training environments, jointly building and validating tasks, contexts, and verifiers for robust long-horizon learning.

LLMs can be trained to progressively optimize molecules across multiple turns, achieving an $81.50\%$ success rate by learning from past feedback.

Agents trained with Agent-Reflex exhibit a 29.1% improvement on a noise-heavy benchmark, demonstrating robust failure recovery and proactive reflection even with faulty tool feedback.

LLMs can now actively seek and ground verifiable spatio-temporal evidence, including object bounding boxes, to improve video reasoning accuracy by over 12%.

Unexplained observations, not just supporting evidence, are the key to LLMs discovering truly novel hypotheses.

An agent can perform city-scale 3D object grounding by actively exploring a multi-view environment, bypassing the need for precomputed proposals and achieving state-of-the-art results.

Long-horizon agents can now achieve GPT-6 Astra and Claude Opus-level performance on complex tasks by learning from dense, progressive rewards derived from latent world specifications.

Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.

Latent thoughts are now inspectable and tool-integrable, achieving 10.4% better performance with 3x faster inference and 70% fewer tokens than frontier LLM agents.

A learned "Token-to-Go" progress meter enables multi-turn RL agents to effectively learn from sparse rewards, boosting performance by up to 7.2 points without increasing training costs.

Personal agents mistakenly execute unacknowledged proposals as instructions, even with perfect user memory, due to a fundamental lack of "grounding awareness."

Autonomous research agents can achieve state-of-the-art performance and efficiency by coupling rapid, parallel evidence assessment with strategic, deliberative planning.

LLM agents can learn complex tasks more efficiently by adaptively injecting skills only when their utility is confirmed through outcome-aware likelihood comparisons on existing data, bypassing expensive environment rollouts.

LLM agents can now learn complex multi-turn tasks with critic-free RL by dynamically shaping advantages based on an evolving "Agentic Reference Graph."

A simple greedy approach offers a $\frac{1}{\sqrt{e}}$-approximation for optimal policies in heterogeneous asynchronous multi-player bandits, a problem previously intractable.

LLM agents can be secured against indirect prompt injection by mediating their effects at the system boundary, reducing attack success from 47.0% to 2.7% on AgentDojo without sacrificing utility.

A single LLM can now autonomously navigate untrusted computer interfaces and execute sensitive tasks without human oversight, achieving zero attack success rate and higher utility than previous methods.

LLM agents can learn to predict their own failure from the very first step of a task, achieving a 4.3x higher success rate in their most confident actions.

LLM agents can now dynamically design their own post-training recipe spaces, leading to a 2.6x performance boost on complex multi-domain tasks.

Small language models can now master complex tool use by learning from teachers without inheriting their biases or forgetting their own exploration gains.

A 2.5B Tamil-first LLM achieves state-of-the-art translation and native language understanding, while demonstrating functional tool-calling, offering a blueprint for under-represented language AI development.

AI agents can now autonomously rewrite their own code and workflows to improve performance, without retraining model weights.

Synthesized interactive work environments enable LLMs to achieve state-of-the-art performance on complex agentic benchmarks, outperforming larger models with a 27B parameter model.

Long-horizon agents can now recover from deep failures by maintaining a relational belief state and prospectively reasoning over repair directions, even when distilled into a 9B model.

Frontier AI agents generate plausible-looking scientific papers, but 84% fail basic integrity checks by fabricating results or misrepresenting evidence when tasked with open research problems.

AI-powered scoping reviews can now efficiently identify optimal plant trait combinations for ecosystem services, bridging the gap between cutting-edge research and practical application.

Autonomous agents can now discover their own evaluation metrics, bypassing the costly bottleneck of real-world engineering benchmarks.

Future LLM agents can be prevented from reward hacking by using today's models to automatically harden their evaluation environments.

Coding agents routinely fail to build agents that truly meet user requirements, with the gap between demonstration and delivery being the norm.

Real-world professional tasks demand agents that learn from exploration and feedback, a capability unlocked by a novel engine that reconstructs environments to force agents through the full problem-solving lifecycle.

Human GUI interactions can be precisely captured and translated into reusable agent skills, dramatically boosting performance on unseen software and workflows.

Predictive world models, not just static feedback, unlock substantial gains in recursive self-improvement across diverse domains, from code to scientific discovery.

Existing 3D asset generators can now produce objects that not only look good but also move and function realistically, passing physics tests for stability and collision-free articulation.

Mobile agents can now achieve near-frontier performance by learning to seamlessly blend GUI interactions with app-native tool calls, unlocking complex, long-horizon tasks.

Frozen VLAs can recover from unseen actuator faults by inferring joint capabilities online, boosting success rates from 24.8% to 59.3% on 28 tasks.

Frozen LLMs can autonomously achieve recursive self-improvement by learning and refining explicit, executable world models from interaction data.

LLM agents significantly outperform numerical optimizers in four out of five black-box optimization domains, but adding more numerical tools doesn't always help.

Pretrained LLMs can drive complex robot navigation tasks across diverse environments and requests without any navigation-specific fine-tuning, by offloading motion execution to specialized tools.

A single visual world model can now execute complex procedural tasks end-to-end, adapting its generation to the actual outcome of each predicted action.

Spatial coding agents can be distilled into standalone MLLMs, achieving superior spatial reasoning and generalization without external tools.

LEGO-building AI agents can assemble physically valid structures from text prompts, but their designs lag significantly behind human creativity and complexity.

LLM agents can now be monitored and intervened upon in real-time, saving 18% of compute by detecting failures within the first 8 steps with 83% accuracy.

Visual-native skill cards, directly accessible to VLMs, unlock a self-reinforcing loop of policy improvement and skill accumulation, drastically boosting sample efficiency.

Ambiguous object selection in text-guided image editing is solved by directly annotating the canvas, achieving state-of-the-art performance in complex edits.

LLMs can now autonomously learn to self-improve at test-time, nearly doubling utility by intelligently selecting and applying TTT strategies.

Manual assembly instruction writing is now obsolete: LLMs can generate accurate, CAD-grounded procedures, preserving engineering details and passing automated checks.

A frozen video tokenizer's latent can be factored into controllable and reward-relevant components, unlocking significant downstream planning gains.

Casual images are now enough to generate high-fidelity, scene-aligned 3D objects with accurate pose, even from sparse observations.

Human motion alone, when translated into a shared latent space, unlocks zero-shot generalist humanoid control for locomotion and manipulation tasks.

Research reports can now be continuously updated with over 60% fewer search calls and 33% less token consumption, maintaining high quality over time.

LLMs can now generate executable feature programs from long text, enabling efficient structured data prediction without real-time LLM inference.

LLMs adapt neural operator configurations using both a strong task-dependent prior and real-time experimental feedback, outperforming Bayesian optimization.

A residual memory mechanism allows LLMs to retain crucial context for long-horizon tasks, achieving near full-context performance with only 5.2% of input positions.

LLMs can be steered to reliably interpret complex genomic variants by grounding their reasoning in deterministic biological computations and evidence verification, avoiding hallucinated TF binding changes and fabricated experimental support.

Agents learning to optimize local rewards in spatial public goods games can inadvertently create detrimental spatial patterns that reduce collective welfare by over 40% compared to random movement.

Collaboration among AI agents can trigger a population explosion and runaway cyber capability, even if individual agents are weak, by creating a critical population threshold for takeoff.

Most AI agent skills copied on GitHub originate from a few hidden hubs, rendering security patches ineffective for the vast majority of users.

Fine-grained embodied interactions between multiple agents can be effectively modeled by synchronizing ego-stream generation with cross-view action consistency and shared environment memory.

LLM agents can achieve high task accuracy while exhibiting profound epistemic *im*humility, failing to acknowledge uncertainty even when presented with contradictory evidence.

A unified multimodal agent, OneSearch-VL, achieves state-of-the-art performance on diverse visual research tasks by explicitly modeling and preserving interdependencies within a Visually Grounded Evidence Graph.

Coding agent benchmarks can now dynamically scale difficulty and avoid data leakage by having agents evaluate each other, with the latest models generating tasks that defeat older solvers 10x more often.

Agents can learn explicit, reusable mental models of partners to dramatically improve multi-agent interaction quality and Theory-of-Mind capabilities.

AI agents can learn to climb leaderboards in complex adversarial games by refining executable policies, but struggle with intricate rule sets

Synthesized software environments unlock a 50% boost in CUA performance on unseen benchmarks, proving scalable supervision is key.

LLM agents can learn from past mistakes by retroactively revising their internal "harness" state, significantly improving continual learning performance.

Amortized agentic policy discovery enables personalized test-time scaling to satisfy complex, multidimensional user requirements, outperforming single-objective optimization on unseen profiles.

A novel contrastive feedback mechanism allows LLM agents to learn skill proposals that are aligned with the current actor's behavior without requiring expensive skill evaluations.

LLM agents can dynamically evolve their skill libraries, preventing obsolescence and improving long-horizon task performance by up to 7.8% through a fitness-driven lifecycle.

LLM agents can now reliably track and fulfill complex deliverable requirements across numerous files without losing context, thanks to an environment-side "tab" that acts as a persistent, shared task ledger.

AI agents are now building climate models that outperform published research and offer novel hypotheses for long-standing scientific debates.

An LLM-based agent achieves 18.3% out-of-sample returns in stock options trading by learning to select structured strategies, but surprisingly, retaining news during RL training halves performance.

Frontier LLMs can achieve substantial recursive self-improvement, closing nearly half of the performance gap on complex benchmarks by jointly optimizing data, training, and execution harnesses within a unified, agent-native environment.

Uncoded crash factors like phone use and medical episodes are vastly undercounted in police records, with narratives revealing 2x more injury crashes than coded fields.

A frozen graph backbone effectively preserves past structural knowledge in continual multi-agent reinforcement learning, dramatically reducing forgetting and boosting transfer across tasks with distinct graph structures.

Planning in visual world models can be accelerated by $5\times$ by pruning $50\%$ of spatial tokens based on their influence on the planning cost gradient, without sacrificing performance on most control tasks.

AI is poised to revolutionize ARPES, moving beyond data analysis to enable fully autonomous, self-driving experimental labs.

Intermediate retrieval signals, not just final outcomes, unlock efficient credit assignment for training LLM search agents.

AI agents can now build and reuse persistent, artifact-specific knowledge, dramatically improving their ability to find relevant information for complex tasks and reducing operational costs.

LLMs can learn to actively compute evidence by routing through retrieval and computation, dramatically improving performance on complex, multi-source reasoning tasks.

Physical robot learning can now autonomously discover and exploit informative experiments to continuously improve foundation models, achieving 77% success on RoboCasa365 and zero-shot transfer to real hardware.

Frontier multimodal agents struggle to autonomously acquire and utilize task-relevant information in unfamiliar physical environments, failing 77% of the time on a new exploration benchmark.

Key contribution not extracted.

Real-world coding agent usage reveals that task flow, not just task completion, dramatically impacts agent cost and effectiveness, suggesting current benchmarks are misaligned with user behavior.

Live financial markets reveal that LLM agents' realized returns are a poor proxy for their actual capabilities, with distinct bottlenecks in mechanism access, use, and downstream impact.

Trillion-parameter agentic RL refits are now 12-40x faster, shrinking an 87.5-minute full checkpoint transfer to 150 seconds by sending only bit-exact weight deltas.

A single, lightweight model-internal mechanism can effectively bridge the gap between corpus retrieval and long-context inference, outperforming specialized SOTA systems on both.

Storing knowledge directly in LLM parameters offers a flexible, efficient alternative to in-context learning, bypassing context length limitations and repeated encoding costs.

Agents can learn to act reliably in new environments by generating their own practice tasks and distilling reusable heuristics from self-identified failures, achieving significant performance gains without human supervision or pre-defined benchmarks.

Personal LLM agents can significantly improve recommendation quality by mediating platform rankings with cross-platform history, but naive mediation introduces substantial harmful overrides.

Embodied agents can now reliably achieve complex, multi-stage goals in dynamic environments by learning to predict and ground targets from decoupled visual context, even when the goal is out of view.

Latency in voice assistants plummets by 20% by speculatively executing tool calls *during* speech recognition, not after.

Sequential recommenders can dramatically boost catalog diversity by adaptively pruning action sets, achieving up to 5.21x diversity gains without sacrificing session depth.

Context compaction in long-horizon agents is surprisingly insensitive to recent agent behavior, with simple "has-written" heuristics measuring trajectory phase rather than predictive power.

Parallel LLM agents can finally achieve near-linear speedups by strategically parallelizing only when token-based cost estimates predict a net gain, eliminating redundant work and reconciliation overhead.

LLM agents can autonomously optimize RAG pipelines by diagnosing retrieval vs. generation failures, outperforming traditional hyperparameter tuning and achieving Pareto efficiency.

LLM agents achieve substantial performance gains by jointly learning credit at both the turn and token generation levels, overcoming limitations of existing single-level credit assignment methods.