Search papers, labs, and topics across Lattice.
The paper introduces pigzpp, a reimagined C++23 library that transforms the traditional pigz parallel gzip utility into a thread-safe, embeddable library suitable for multiple programming languages. This new design not only maintains compatibility with existing gzip standards but also enhances performance, achieving up to 2.1 times the throughput of pigz and 16 times that of Python's gzip in specific configurations. Additionally, pigzpp includes features like multi-entry ZIP archives and a fast PNG codec, showcasing a significant leap in compression efficiency and usability across platforms.
pigzpp achieves up to 16 times faster compression than Python's gzip while maintaining full compatibility with existing gzip standards.
pigz is a widely deployed parallel gzip utility, but its process-global mutable state means that it was not designed as a reentrant, directly embeddable library. pigzpp is a from-scratch C++23 rewrite that turns the design into a thread-safe library with one accelerated DEFLATE core exposed to C++, Python, WebAssembly, Go, and Rust. As application-level conveniences built on that same core, it also provides multi-entry ZIP archives and a fast PNG codec. Its portable zlib-ng backend preserves gzip/zlib's ratio; its Intel ISA-L backend is an x86-64-only fast path that produces about 10% larger output at level 6 on our text corpus. In that configuration, zlib-ng reaches 2.1 times the CLI throughput of pigz and 16 times that of Python's in-memory gzip; ISA-L reaches 8.5 times and 50 times, respectively. Outputs remain standards-compliant and cross-decode with gzip, pigz, and unzip. The result is a self-contained, multi-platform compression stack and a case study in AI-assisted modernization under automated compatibility tests.
Microsoft Research