Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents a systematic literature review of optimization strategies for cloud-based data pipeline infrastructures, focusing on cost, speed, and resource utilization. It synthesizes existing research into a theory of optimization goals, categorized by dimensions like single vs. multi-cloud and batch vs. stream processing. The review identifies gaps in current research, specifically the limited exploration of multi-tenant environments and the absence of industry evaluations, thereby charting a course for future investigations.
Cloud data pipeline optimization is surprisingly underexplored in multi-tenant environments and lacks real-world industry validation, leaving significant performance gains on the table.
Cloud infrastructure supports the efficient operation of data pipelines regarding requirements like cost, speed, and resource utilization. We present an integrated view of optimization opportunities for cloud-based data pipelines by conducting a systematic review of existing literature on optimization approaches to cloud infrastructure performance for data pipelines. Our study contributes a theory of optimization goals like minimizing cost, reducing execution time, and cost-makespan trade-offs, consisting of dimensions such as single vs. multi-cloud, batch vs. stream processing, etc. We highlight gaps in primary research, including the underexploration of multi-tenant environments and lack of industry evaluation, and suggest directions for future research.