Search papers, labs, and topics across Lattice.
This paper introduces BiJuTy, an interactive lifecycle manager and performance assessment utility designed for executing big data workflows on high-performance computing (HPC) systems within the Jupyter ecosystem. By providing a user-friendly interface that streamlines cluster management and performance evaluation, BiJuTy significantly lowers the barrier to entry for both novice and experienced users. The framework allows for seamless management of multiple clusters and simplifies the optimization process through the collection of performance metrics, demonstrating its effectiveness with a practical example of big data application optimization.
BiJuTy transforms the way users interact with HPC systems, enabling effortless big data processing workflows directly within Jupyter.
The increasing demand for data processing has created a pressing need for access to high-performance computing (HPC) systems. Nevertheless, leveraging these systems to execute complex big data processing workflows remains a significant challenge, especially for beginners. This work presents BiJuTy, a solution designed to bridge the accessibility gap for big data workflows on HPC systems within the Jupyter ecosystem. By providing an interactive and user-friendly interface, BiJuTy simplifies cluster lifecycle management and performance assessment, making it more accessible on HPC systems to beginners and experienced users alike. The solution is presented as an interactive interface that guides the user through the entire process, from setting up the cluster configuration to carrying out initial performance assessments. Additionally, the framework enables seamless management of multiple clusters directly within the Jupyter Notebook interface, eliminating the need to switch outside of working environment. The collection of performance metrics from various sources further simplifies the optimization workflow. Furthermore, an illustrative example is provided to demonstrate how BiJuTy can be deployed to optimize the performance of a big data processing application. This example showcases how the entire big data processing lifecycle can be iteratively executed and optimized in just a few clicks, helping to reach the goal of optimization easily and interactively. By facilitating such workflows, this work contributes in bringing the field of big data computing and high-performance computing one step closer to the goal of seamless interaction and usability.
