Search papers, labs, and topics across Lattice.
This paper addresses the challenge of workflow composition in High-Throughput Computing (HTC) environments by introducing a simulation framework that analyzes the relationship between taskset granularity and system-level constraints. Through extensive exploration of a high-dimensional parameter space, the authors reveal that hybrid composition strategies can significantly enhance performance, achieving up to a 3.8x increase in throughput and a 14.9x reduction in network overhead. Additionally, they propose a multi-metric objective function that aids in optimizing workflows, providing a robust foundation for automated workflow synthesis in distributed systems.
Hybrid workflow strategies can boost throughput by 3.8x while slashing network overhead by nearly 15x, revolutionizing data processing efficiency.
High-Throughput Computing (HTC) environments tailored for high-concurrency resource efficiency require sophisticated orchestration to manage petabyte-scale data across heterogeneous resources. A critical but often overlooked challenge is workflow composition: the strategic grouping of tasksets within a Directed Acyclic Graph (DAG) to mitigate execution overhead while maximizing resource utilization. This paper presents a novel simulation framework for characterizing the interplay between taskset granularity and system-level constraints (e.g., job latency, failure rate, throughput, and I/O bandwidth). By exploring a high-dimensional parameter space, we quantify the performance sensitivity of diverse workflow topologies. Our results demonstrate that hybrid composition strategies, which dynamically balance taskset independence with execution grouping, can yield up to 3.8x throughput increase and a 14.9x reduction in network overhead. We further propose a multi-metric objective function that enables policy-driven optimization, allowing system architects to navigate the Pareto frontier between throughput, I/O cost, and CPU efficiency. These findings provide a rigorous foundation for automated workflow synthesis in distributed systems, offering a scalable model for next-generation scientific pipelines. All artifacts are publicly available.
