Search papers, labs, and topics across Lattice.
This study evaluates the performance of small language models in a Retrieval-Augmented Generation (RAG) system, highlighting their capability to operate efficiently on-device without GPU support. By benchmarking these models across various datasets and question types, the research reveals that compact models can achieve competitive results in generation tasks. The findings underscore the potential of small language models to maintain relevance in AI applications despite the dominance of larger counterparts.
Small language models can effectively power on-device RAG systems without the need for GPU resources, challenging the assumption that bigger is always better.
While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention. In this study, we investigate how smaller language models perform during the generation stage within a Retrieval-Augmented Generation (RAG) system. To benchmark these models effectively, we utilised both open-source and proprietary datasets covering diverse subject areas and question types. Our findings demonstrate that a RAG system with small language models can be executed directly on-device without requiring any GPU hardware within a reasonable time. The experimental code and links to the supplementary materials can be accessed through the GitHub repository: https://github.com/SibNN/SLM-RAG-EVAL.
