Search papers, labs, and topics across Lattice.
This paper introduces DEFRAG, a decentralized edge collaboration system designed for retrieval-augmented generation (RAG) that enhances the deployment of small language models (SLMs) on resource-constrained edge devices. By optimizing both retrieval through compressed knowledge graphs and adaptive generation using a novel optimizer, DEFRAG significantly narrows the accuracy gap between SLMs and cloud-based large language models (LLMs). The implementation shows that DEFRAG can reduce costs by up to 98.4% while increasing peak throughput by up to 97.8%, making LLM services more accessible and efficient at the edge.
DEFRAG narrows the accuracy gap between small and large language models while slashing costs by up to 98.4%, revolutionizing edge-based AI deployment.
The rapid advancement of large language models (LLMs) has increased demand for scalable and cost-effective deployment, especially for mobile and edge devices. Cloud-hosted LLMs are powerful but expensive and difficult to scale due to vendor lock-in and high resource needs, resulting in high expenses and unstable performance under load. Recent efforts focus on deploying small language models (SLMs), distilled or pruned from LLMs, on resource-constrained edge devices to reduce costs and improve scalability. However, edge-based SLMs face limited knowledge coverage and notable accuracy gap compared to cloud-based LLMs. To address this, we present DEFRAG, a decentralized edge collaboration system for retrieval-augmented generation (RAG) that optimizes both retrieval and generation across heterogeneous edge devices. For retrieval, DEFRAG compresses and shares knowledge graphs, using hybrid retrieval to expand knowledge coverage. For generation, DEFRAG introduces an optimizer that adaptively selects SLMs and RAG parameters per query, balancing accuracy and cost. We implement DEFRAG on a heterogeneous edge testbed and evaluate it on benchmark QA datasets. We also test it under mobile route stress, non-uniform data placement, and a domain-specific QA workload. The results show that DEFRAG maintains stable service quality and cost efficiency under these broader settings. Results show that DEFRAG narrows the SLM-LLM accuracy gap, while reducing cost by up to 98.4% and increasing peak throughput by up to 97.8% over centralized services. These findings demonstrate the potential of DEFRAG for democratized LLM services at the edge.
