Search papers, labs, and topics across Lattice.
This paper introduces CHAMB-GA, a containerized microservice-based framework designed to enhance the scalability and flexibility of genetic algorithms (GAs) with embedded simulations. By enabling seamless transitions from personal computers to distributed clusters, the framework addresses the limitations of existing tools that are often rigid and hardware-bound. The results demonstrate CHAMB-GA's ability to efficiently scale to over 3,500 CPU cores with minimal overhead, showcasing its effectiveness in complex applications like HVDC line optimization in power grids.
CHAMB-GA scales genetic algorithms across distributed systems with minimal overhead, revolutionizing how we approach computationally intensive optimization tasks.
Metaheuristic-based global optimization with embedded, long-running simulations is a computationally expensive process. To support various stages of development and execution, a seamless transition from personal computers to distributed clusters is desired, enabling execution across all computational scales. However, existing tool chains are often characterized by rigidity and hardware-bound constraints, which impede scalability and the integration of complex simulations. Bridging this gap, we present a containerized HPC scalable microservice-based framework for genetic algorithms with embedded simulations (CHAMB-GA). The deployment of the framework scales consistently across cloud infrastructure via container orchestration and HPC clusters via batch-scheduled parallel execution. Users provide the GA operators and simulation backend separately. The framework is designed to run these components in a distributed and decoupled manner, mapped to separate hardware. This approach ensures that the fitness evaluation and genetic operations are not managed within the same process and are utilizing distinct parts of the compute infrastructure. A central message broker coordinates asynchronous manager-worker communication between microservices, thereby parallelizing evolutionary operations and fitness evaluations. We demonstrate CHAMB-GA's scalability, portability, and reproducibility, while facilitating the integration of external tools and complex simulations on benchmark and powerflow problems. The capabilities of CHAMB-GA are validated in a two-part approach: (i) a benchmark study demonstrating minimal overhead while scaling to over 3,500 CPU cores, and (ii) a dispatch optimization of High Voltage Direct Current (HVDC) lines in the German transmission grid, showing seamless migration from Kubernetes to SLURM, combined horizontal and vertical scaling, and integration of multi-stage workflows.
