Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
To address the petabyte-scale data bottlenecks anticipated at the High-Luminosity LHC, the authors built and stress-tested end-to-end CMS analysis pipelines built on the modern scientific Python ecosystem and HEP-specific columnar tools. The IRIS-HEP Integration Challenge was deployed as an end-to-end benchmark to rigorously evaluate toolchain interoperability across data skimming, columnar transformation, machine learning inference, and statistical visualization. The implementation validates that distributed, columnar architectures paired with data-delivery systems like ServiceX can support complex multi-stage physics workflows while isolating key latency bottlenecks in emerging prototype services.
Handling the High-Luminosity LHC's massive data explosion requires replacing legacy monolithic HEP frameworks with an interoperable, Pythonic columnar software stack tightly coupled with distributed ML infrastructure.
The upcoming High-Luminosity Large Hadron Collider (HL-LHC) at CERN will deliver an unprecedented volume of data for High Energy Physics (HEP). This wealth of information offers significant opportunities for scientific discovery, but its scale challenges traditional analysis workflows. In this paper, we present CMS analysis pipelines being developed to meet HL-LHC demands. These pipelines build on the broader scientific Python ecosystem, complemented by solutions specifically designed for HEP. A central focus of this work is the Integration Challenge, an IRIS-HEP led effort aimed at assessing the readiness of developed software stack to be used in real world physics analysis and improving the readiness of analysis facilities for the HL-LHC era. The Integration Challenge acts as an end-to-end integration test: by implementing a complete physics analysis pipeline, it evaluates tool interoperability and the overall user experience for analysts. The current pipeline includes columnar data processing, machine learning, statistical inference, and visualization tasks covering a variety of CMS analysis scenarios. In addition, the Integration Challenge explores efficient strategies for delivering skimmed data using diverse tools and data formats, as well as evaluating the ServiceX data-delivery system for HEP analyses. Throughout the testing phase, we also investigated several prototype services, such as histogram-as-a-service capabilities, along with other emerging services that may support future HL-LHC analysis workflows.