Search papers, labs, and topics across Lattice.
This paper explores the implementation of a resilient, MPI-based key-value store using passive target MPI RMA functions and the ULFM extension for failure detection, intended for use in resilient task-based runtime systems. The authors encountered difficulties due to incomplete ULFM functionalities for RMA, hindering the straightforward implementation of failure recovery mechanisms. They detail their experiences, highlighting missing functionalities and presenting a workaround to achieve resilience in the key-value store.
Implementing resilient, MPI-based key-value stores with RMA and ULFM is harder than it looks due to missing functionalities, requiring workarounds to achieve fault tolerance.
As hardware failures such as node losses become increasingly common, MPI programmers may want to save vulnerable data in a resilient store. While third-party storage solutions such as Redis or the Hazelcast IMap exist, a tailored, MPI-based store may be easier to integrate and can be optimized for particular application needs. This paper considers the implementation of such a store, which is intended as a component in a resilient task-based runtime system written in MPI. The store holds redundant data copies as key-value pairs in the main memories of multiple processes. Since store access operations, such as reads and writes, are naturally one-sided, we implemented the store with passive target MPI RMA functions. Process aborts are detected with the user-level failure mitigation (ULFM) extension of Open MPI. After failures, the program recovers on the surviving processes and continues with the intact data copies. Our implementation proved difficult, since several proposed ULFM functionalities for RMA have not yet been implemented. Even assuming their existence, we think that the programming task could be simplified. This paper describes our experiences, lists functionalities that we missed, and explains a workaround that we adopted in our implementation.
