Search papers, labs, and topics across Lattice.
This paper introduces NIXT, a tool designed to enhance the observability of collective communication during large model training by exporting actionable insights from the Nvidia Collective Communication Library (NCCL) Inspector. By addressing the challenge of interpreting the extensive data generated by NCCL Inspector, NIXT enables researchers to identify performance variations and conduct root cause analysis effectively. A case study involving the pretraining of the Nemotron-4 LLM on a large GPU cluster illustrates NIXT's capabilities in analyzing communication dynamics and attributing performance discrepancies to specific factors like stragglers.
NIXT transforms NCCL Inspector's data overload into clear, actionable insights, enabling researchers to pinpoint performance issues in large-scale model training.
As machine learning workloads scale, it is increasingly important to gain more observability into the performance of collective communication to easily identify performance vari- ations and accelerate root cause identification. Towards this goal, the Nvidia Collective Communication Library (NCCL) introduced NCCL Inspector, a profiler plugin that provides lightweight and continuous reporting of NCCL communication performance statistics. However, the large volume of data collected by NCCL Inspector can be difficult to assess and to extract actionable insights from. This paper presents NIXT, a NCCL Inspector Exporter Tool that improves the observability of collective communication by providing readily accessible analysis and actionable insights from NCCL Inspector profiling. To highlight the benefits of our Exporter Tool, we present a case study of Nemotron-4 LLM pretraining on an Nvidia H100 GPU cluster with up to 2,048 GPUs, demonstrate observability into how communication phases change with ML parallelism and GPU scale, and perform attribution of performance variation and root cause analysis of stragglers.
