Search papers, labs, and topics across Lattice.
This paper introduces a full-device resident GPU decode pipeline for LZ77 compression, achieving up to 260GB/s on FASTQ files, significantly enhancing genomic data processing speeds. It features position-invariant random access that allows arbitrary reads to be decoded in just 0.362ms, outperforming existing CPU tools like samtools by a factor of six. Additionally, a novel range-decode strategy enables high-speed decoding of large genomes without exceeding VRAM limits, ensuring bit-perfect results throughout the process.
Achieving 260GB/s decoding speeds on genomic data, this work revolutionizes how we access and process massive biological datasets.
Genomic archives grow faster than decompression keeps up: the European Nucleotide Archive holds tens of petabytes of fastq.gz, and gzip is fundamentally sequential. GPU decompressors (nvCOMP DEFLATE at ~50GB/s on A100) decode whole files with no random access; CPU genomic tools (CRAM, samtools) support region seeks but only at CPU speed. We extend ACEAPEX, an absolute-offset parallel LZ77 codec included in the official lzbench 2.3 release, with three contributions absent from our prior work. First, a full device-resident GPU decode pipeline (entropy and match resolution both on-device) reaching up to 260GB/s on FASTQ, closing the match-phase-only gap of the earlier paper. Second, position-invariant random access with a compact coordinate index: an arbitrary read decodes in 0.362ms, ~6x faster than warm samtools faidx, with a read-to-block index 6.3x smaller than a .fai. Third, a range-decode strategy that decouples output size from VRAM, sustaining 165.7GB/s on a 50GB genome where whole-file decode runs out of memory. All results are bit-perfect. We also measure Meta's open DietGPU ANS on H100 at 592GB/s decode, faster than the proprietary entropy stage we currently use, showing a fully open high-throughput stack is viable. Code is MIT-licensed.
