Search papers, labs, and topics across Lattice.
This paper introduces a framework leveraging the Project Panama Vector API for memory-efficient predicate evaluation by directly operating on bit-sliced, compressed data streams. By transforming standard row-oriented data into parallel bit-planes, the method allows for SIMD-based evaluation of complex filters without the need for prior decompression, significantly reducing memory overhead. Benchmarks reveal up to an 8x reduction in memory usage and speedups of 2.4-10.8x in throughput for filter-heavy queries compared to traditional Java collections.
Achieving up to 8x memory savings and over 10x speedup in query performance, this framework revolutionizes how we handle data-intensive operations in distributed systems.
Traditional Java object models introduce significant memory overhead due to object headers and internal padding, often leading to performance bottlenecks in data-intensive distributed systems. This paper presents a novel framework that utilizes the Project Panama Vector API to perform predicate evaluation directly over bit-sliced, compressed data streams. By transposing standard row-oriented data into parallel bit-planes, we demonstrate a mechanism to evaluate complex filters using SIMD (Single Instruction, Multiple Data) instructions without requiring prior decompression. The framework supports integers, longs (timestamps), doubles (via IEEE 754 order-preserving transformation), and strings (via dictionary encoding). Our benchmarks indicate a reduction in memory footprint by up to 8x while maintaining or exceeding the throughput of uncompressed standard Java collections. End-to-end evaluation on TPCDS-modeled data at 50M rows demonstrates 2.4-10.8x speedup over scalar scans across five representative filter-heavy query patterns, with extended type benchmarks on TPCDS columns showing 1.5-43x speedups for timestamps, decimals, and dictionary-encoded strings.
