Search papers, labs, and topics across Lattice.
This paper systematically analyzes retry policies in distributed systems, focusing on their collective behavior during partial failures and introducing the retry amplification factor (RAF) to quantify the increased request volume caused by retries. Through an examination of 200 open-source Python microservice projects, the authors find that while explicit retry logic is present in 11.5% of projects, the true prevalence is closer to 41%, with many configurations lacking essential backoff mechanisms. The study reveals that naive retry strategies can significantly decrease success rates under correlated failures, and proposes Adaptive Retry Budgeting (ARB) as a solution that maintains high success rates while effectively handling transient faults.
Retry policies can inadvertently amplify failures, reducing success rates by over 25% when not designed with system-level considerations in mind.
Retry mechanisms are a standard component of resilient distributed systems, but their collective behavior, when every tier in a call path retries concurrently, is less well understood than the per-client guidance that produced them. This paper introduces the retry amplification factor (RAF), a metric quantifying the additional request volume that retry policies generate during partial failures. In a study of 200 open-source Python microservice projects, explicit retry logic is detected in 11.5%, and an audit of our own false negatives places true prevalence near 41%. Among the projects detected, 60.9% contain at least one configuration without backoff, and after manual verification exactly one of 113 production configurations randomizes its delay. We then evaluate these policies in simulation (n = 100 trials per strategy). Under correlated failures, a naive standard retry policy reduces the success rate from 55.4% to 41.5% relative to performing no retries at all. We catalog five recurring anti-patterns, propose Adaptive Retry Budgeting (ARB), and show that budget-constrained retries maintain success rates close to the no-retry baseline while still recovering from transient faults. These results indicate that retry behavior should be designed as a system-level property rather than configured locally at each call site.
