Search papers, labs, and topics across Lattice.
This paper presents scrydb, a Python library that integrates lexical, semantic, and hybrid search capabilities within SQLite, utilizing its FTS5 extension for lexical search and sqlite-vec for semantic search. The library allows for reranking and fusion of retrieval results, enhancing performance in information retrieval tasks. Evaluations on benchmark datasets reveal that scrydb effectively balances keyword matching and semantic similarity while providing insights into query latency and efficiency trade-offs.
scrydb achieves a powerful blend of lexical and semantic search in SQLite, offering a lightweight yet effective solution for modern information retrieval challenges.
This work introduces scrydb, a Python library that enables lexical, semantic, and hybrid search within SQLite. For lexical search, scrydb leverages SQLite's full-text search extension FTS5. Semantic search builds on sqlite-vec, a SQLite extension for vector search. Furthermore, the library allows users to rerank and fuse retrieval results to combine both lexical and semantic approaches, providing a lightweight solution for downstream tasks in information retrieval (IR) or agentic search. We evaluate scrydb on various IR benchmark datasets and demonstrate its effectiveness in text retrieval based on keyword matching, semantic similarity, and rank fusion. In addition, we provide insights into query latency and the trade-off between efficiency and effectiveness. scrydb is available under the MIT license.
