Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces uCache, a client-side caching solution for high-energy physics data analysis that operates transparently without requiring server-side changes or modifications to existing analysis code. By utilizing local storage to cache only the data actually accessed, uCache significantly reduces the input/output and decompression overhead for subsequent data processing passes. Benchmark results demonstrate that the caching mechanism can accelerate analysis speeds by up to 15.7 times, particularly under conditions of high network load or geographical distance from data sources.
Caching remote data can boost analysis speeds by up to 15.7 times, transforming how physicists handle large datasets.
High-energy physics analyses often process the same data as physicists refine algorithms and test new ideas. With data increasingly read from remote storage, each iteration is subject to network latency and depends on network bandwidth and shared-storage throughput, which can vary substantially under load. We present uCache (xrd-ucache), a transparent client-side cache implemented as an XRootD client plugin that requires neither server-side deployment nor changes to analysis code. It uses local storage on the analysis machine as a cache layer between the network and memory. The cache stores only the data actually read by an analysis. It can also rebuild cached data into a branch-aligned, recompressed form, eliminating most of the input/output and decompression costs of subsequent passes. We benchmark the cache using the Analysis Grand Challenge top quark pair analysis on public CMS Open Data compressed with zlib and LZMA. Filling the cache adds essentially no overhead compared with a direct read. Subsequent passes are 1.6-8.8 times faster from the byte cache and 2.1-15.7 times faster from the recompressed cache. For a typical analysis, a 1 TB cache suffices for datasets of 10-20 TB. The largest improvements occur when the remote data source is heavily loaded or geographically distant.
MIT CSAIL