Search papers, labs, and topics across Lattice.
This paper explores the feasibility of repurposing retired GPUs to create a cost-effective and environmentally sustainable "DumpsterCluster" for serving modern LLMs like LLaMA-70B. By constructing a 128-GPU system using second-hand components, the authors demonstrate that this setup can achieve competitive throughput with significant economic advantages, costing only $22K compared to $600K for a new system. However, the study also highlights critical context dependencies, revealing that while older GPUs can expand AI capacity, they may lead to substantially higher carbon emissions unless paired with low-carbon energy sources.
Repurposing retired GPUs can cut costs dramatically, but without clean energy, it risks quadrupling carbon emissions for LLM inference.
As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can serve modern LLM inference, and under what conditions such repurposing is economically viable and environmentally sustainable. We physically built a 128-GPU DumpsterCluster from scratch using only second-hand components and ran it for one year. At current market prices (\$22K for the DumpsterCluster vs. \$600K for an 8-GPU B200 system), the economic advantages are substantial. Through pipeline-parallel optimizations, our V100 based DumpsterCluster achieves competitive LLaMA-70B throughput, validating production viability. However, our deployment reveals critical context dependencies. Older GPUs consume significantly more energy per token, making total cost of ownership favorable only in regions with inexpensive electricity. Under grid-average carbon intensity, second-hand systems can produce approximately 4x higher total carbon emissions per token for 8B models, and over 40x for 70B models, compared to current-generation hardware. These findings show that GPU afterlife is not universally sustainable - hardware repurposing must be strategically coupled with low carbon energy sources. When deployed in regions with favourable energy economics and clean electricity, second-hand GPUs offer a viable pathway for expanding AI capacity while advancing affordability, energy security, and environmental responsibility.
DeepMind