Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper systematically analyzes retry policies in distributed systems, introducing the retry amplification factor (RAF) to quantify the excess request volume generated during partial failures. Through an examination of 200 open-source Python microservice projects, it reveals that a significant portion of these systems lack effective retry configurations, with 60.9% using at least one policy without backoff. The study demonstrates that naive retry strategies can drastically reduce success rates and proposes Adaptive Retry Budgeting (ARB) as a solution that maintains high success rates while effectively managing retries during transient faults.
Retry strategies can amplify failures, but Adaptive Retry Budgeting can keep success rates near optimal even under stress.
Retry mechanisms are a standard component of resilient distributed systems, but their collective behavior, when every tier in a call path retries concurrently, is less well understood than the per-client guidance that produced them. This paper introduces the retry amplification factor (RAF), a metric quantifying the additional request volume that retry policies generate during partial failures. In a study of 200 open-source Python microservice projects, explicit retry logic is detected in 11.5%, and an audit of our own false negatives places true prevalence near 41%. Among the projects detected, 60.9% contain at least one configuration without backoff, and after manual verification exactly one of 113 production configurations randomizes its delay. We then evaluate these policies in simulation (n = 100 trials per strategy). Under correlated failures, a naive standard retry policy reduces the success rate from 55.4% to 41.5% relative to performing no retries at all. We catalog five recurring anti-patterns, propose Adaptive Retry Budgeting (ARB), and show that budget-constrained retries maintain success rates close to the no-retry baseline while still recovering from transient faults. These results indicate that retry behavior should be designed as a system-level property rather than configured locally at each call site.