Search papers, labs, and topics across Lattice.
FluxSieve introduces a unified architecture for cloud observability that combines pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer into the data ingestion path. This approach uses a scalable multi-pattern matching mechanism for concurrent evaluation and on-the-fly updates of filtering rules. Experiments integrating FluxSieve with Apache Pinot and DuckDB show orders-of-magnitude improvements in query performance with negligible storage and computational overhead.
Achieve orders-of-magnitude faster queries in cloud observability platforms by pushing filtering logic into the data ingestion pipeline, rather than relying solely on analytical data plane query optimization.
Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.
