Search papers, labs, and topics across Lattice.
This paper introduces StreamGuard, a framework designed to enhance the resilience of real-time scientific workflows operating on continuous data streams by addressing hardware faults and performance anomalies. The authors implement two key techniques: a dynamic, asynchronous checkpointing mechanism that allows for uninterrupted computation and a progress-aware load redistribution strategy that optimally reallocates tasks from slower workers. Experimental results indicate that StreamGuard can mitigate the effects of failures and performance issues by up to 6x, while maintaining less than 1% overhead during normal operations.
StreamGuard can reduce the impact of failures in real-time data streams by up to 6x with minimal overhead, revolutionizing resilience in scientific computing.
Real-time scientific workflows operate on continuous data streams and must produce timely, high-quality results despite executing on complex, failure-prone infrastructure. Hardware faults, network disruptions, and performance anomalies caused by resource contention or system heterogeneity can severely degrade performance and violate real-time constraints. We focus on strengthening the resilience of the producer-consumer streaming pattern, a fundamental building block of scientific streaming workflows. We present two complementary techniques: (i) a dynamic, asynchronous, non-blocking checkpointing mechanism that preserves progress without interrupting computation, and (ii) a progress-aware load redistribution strategy that detects slow workers and proactively rebalances tasks. Together, these mechanisms maintain forward progress and balanced execution even in highly error-prone environments. Experimental results show that our approach reduces the impact of failures and performance anomalies by up to 6x, while introducing less than 1% overhead in failure-free execution.
