Search papers, labs, and topics across Lattice.
This paper benchmarks Apache Pulsar, achieving 1.5 million messages per second at 3.88 ms median publish latency on three bare-metal Kubernetes nodes. The study identifies hardware, specifically NVMe journals and network interfaces, as the primary bottleneck for throughput, projecting a path to 15 million msg/s on 15 machines. Through Java Flight Recorder guided root cause analysis, the authors optimized latency by switching to ZGC Generational, dedicating NVMe journals, and addressing a Linux kernel page cache writeback interaction.
Unexplained P99.9 latency spikes in Apache Pulsar could be due to a previously undocumented Linux kernel page cache writeback interaction inside BookKeeper's ForceWriteThread, even with dedicated NVMe drives.
This paper presents two independent contributions for Apache Pulsar practitioners. First, we validate 1,499,947 msg/s at 3.88 ms median publish latency on just three bare-metal Kubernetes nodes running Pulsar 4.0.8 with Java 21 and ZGC Generational garbage collection, and project a hardware-driven path to 15 million msg/s on 15 machines using five independent clusters with key-based partition routing. Hardware selection -- specifically dedicated NVMe journals achieving 0.02 ms fdatasync and 25 Gbps network interfaces -- is the primary determinant of throughput ceiling, not compute or software tuning. Second, we trace the complete latency optimization journey from 213 ms GC spikes and 13-18 ms median publish latency in production to 3.88 ms through Java Flight Recorder guided root cause analysis. Three independent root causes are identified and resolved: G1GC pauses eliminated by switching to ZGC Generational; journal fdatasync latency reduced from 5.1 ms to 0.02 ms through NVMe journal dedication; and a previously undocumented Linux kernel page cache writeback interaction inside BookKeeper's ForceWriteThread that degrades fdatasync from under 1 ms to 15-22 ms even across physically separate NVMe drives sharing the kernel block layer. This finding is undocumented in official Apache Pulsar and BookKeeper documentation and is relevant to all Pulsar operators experiencing unexplained P99.9 latency spikes. The combined optimizations achieve a 4.7x latency improvement at 50x higher throughput.
