Search papers, labs, and topics across Lattice.
This paper introduces llmmas-otel, a novel tool that integrates OpenTelemetry-based distributed tracing with fault injection specifically for LLM-based multi-agent systems in software engineering. The tool enhances the observability of agent executions by providing trace-aligned telemetry across various workflow phases and enabling targeted fault injection, thereby facilitating the comparison of baseline and faulty executions. Initial validation demonstrates its effectiveness in improving the inspection and debugging processes of LLM-based systems, which are crucial for reliable software engineering applications.
Fault injection in LLM-based multi-agent systems can now be systematically analyzed, revealing how failures propagate through complex workflows.
Large Language Model-based multi-agent systems are increasingly explored for software engineering tasks, but they remain difficult to inspect, debug, and evaluate under controlled failures. We present llmmas-otel, a lightweight and framework-agnostic tool that combines OpenTelemetrybased distributed tracing with fault injection for LLM-based multi-agent systems in software engineering workflows. The tool instruments agent executions with trace-aligned telemetry across workflow phases, agent steps, inter-agent communication, tool calls, and LLM invocations, and supports targeted fault injection at selected interaction points. This makes it possible to compare baseline and faulty executions in a reproducible way and inspect the effects through aligned traces and run artifacts. We describe the motivation, architecture, implementation, current capabilities, and initial validation of the tool on a minimal demo workflow and a real LLM-based multi-agent system for software development. A screencast is available at: $\langle$ https://youtu.be/tyQh2xO-flo $\rangle$
