Search papers, labs, and topics across Lattice.
This paper introduces an experiment-driven approach for automatically tuning the configuration of stream processing systems, specifically Kafka Streams, in cloud-native Kubernetes environments. The approach combines Latin Hypercube Sampling, Simulated Annealing, and Hill Climbing to efficiently explore the configuration space and identify optimal settings. Experiments demonstrate a throughput improvement of up to 23% compared to default configurations, highlighting the effectiveness of Latin Hypercube Sampling with early termination and Simulated Annealing.
Achieve up to 23% higher throughput in Kafka Streams by automating configuration tuning with a novel experiment-driven approach.
Configuring stream processing systems for efficient performance, especially in cloud-native deployments, is a challenging and largely manual task. We present an experiment-driven approach for automated configuration optimization that combines three phases: Latin Hypercube Sampling for initial exploration, Simulated Annealing for guided stochastic search, and Hill Climbing for local refinement. The workflow is integrated with the cloud-native Theodolite benchmarking framework, enabling automated experiment orchestration on Kubernetes and early termination of underperforming configurations. In an experimental evaluation with Kafka Streams and a Kubernetes-based cloud testbed, our approach identifies configurations that improve throughput by up to 23% over the default. The results indicate that Latin Hypercube Sampling with early termination and Simulated Annealing are particularly effective in navigating the configuration space, whereas additional fine-tuning via Hill Climbing yields limited benefits.
