Search papers, labs, and topics across Lattice.
This paper explores the feasibility of using quantized, open-weight language models for integration with large databases, revealing that these models can achieve comparable or superior accuracy to proprietary APIs while significantly reducing costs and latency. By implementing these models within the BlendSQL v0.1.0 framework, the authors demonstrate a remarkable 390x reduction in overall costs and a 3.8x reduction in latency compared to traditional closed-source solutions. This challenges the prevailing belief that only proprietary language models can effectively enhance database operations, paving the way for more accessible and cost-effective AI solutions.
Open-weight language models can outperform costly proprietary APIs, slashing expenses by 390x and latency by 3.8x in database integrations.
Language model systems built around proprietary APIs often operate on a token-based cost model. This becomes prohibitively expensive in the context of large databases, where LM-enhanced relational operators can incur costs exceeding $10,000 for a single set of experiments, hindering thorough research and practical deployment. In this paper, we demonstrate that quantized, open-weight models running locally on just 16GB of VRAM can match or exceed the accuracy of closed-source counterparts at lower latency and a fraction of the price, challenging the prevailing assumption that closed-source LM APIs are necessary for effective LM-database integration. We present and analyze the key system optimizations required to efficiently deploy these open-weight models within an LM-DB system. By integrating these local models into the BlendSQL v0.1.0 framework, we demonstrate a 390x reduction in overall costs and 3.8x reduction in latency compared to a proprietary LM API. We make our code available at https://github.com/CapitalOne-Research/play-by-the-type-rules/tree/main/sembench.
