Search papers, labs, and topics across Lattice.
This paper introduces BlobShuffle, a novel method for cost-effective data shuffling in stream processing systems that utilizes cloud object storage to minimize network traffic and operational burdens. By batching records and storing them in object storage, BlobShuffle significantly reduces shuffling costs by over 40x while maintaining low latency and high throughput. Experimental results demonstrate its scalability, capable of processing more than 2 GiB/s without hitting performance limits, making it a viable solution for large-scale stateful workloads.
BlobShuffle slashes shuffling costs by over 40x while keeping latency under 2 seconds, revolutionizing how we handle data in stream processing.
Shuffling or repartitioning data streams is an essential operation of state-of-the-art stream processing frameworks to support stateful workloads in a large-scale, distributed setting. In today's cloud deployments, however, shuffling can become a major cost driver due to substantial network traffic across multiple availability zones (AZs) as well as an operational burden when operating a high-throughput, strongly consistent messaging backbone at scale. We present BlobShuffle, a novel approach to cost-effective shuffling for stream processing systems that leverages cloud object storage as an intermediate exchange layer. Instead of sending all shuffled records directly, BlobShuffle groups records into batches, stores these batches in cloud object storage, and forwards only compact notifications. Downstream operators use these notifications to retrieve the relevant batches and extract the corresponding records. BlobShuffle balances cost efficiency and latency through configurable batching and a distributed caching mechanism. BlobShuffle is implemented as an add-on for Kafka Streams that requires only minimal code changes to existing applications, leaves Kafka and the underlying infrastructure unmodified, and preserves Kafka Streams'consistency and correctness guarantees. In a large-scale experimental evaluation on a Kubernetes-based AWS deployment, we show that BlobShuffle can reduce shuffling costs by more than 40x compared to native Kafka Streams shuffling while keeping the 95th percentile shuffle latency below 2 seconds. Moreover, it scales to processing more than 2 GiB/s without encountering a scalability limit in our experiments, indicating that BlobShuffle can economically support shuffle-intensive workloads at large scale.
