Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
HAKARI-Bench is introduced as a lightweight benchmark designed to facilitate the comparison of various retrieval architectures and efficiency settings under consistent conditions. By reconstructing existing retrieval suites into compact datasets, it allows for model-agnostic evaluations across five retrieval families and their efficiency variants, covering 35 benchmarks and 551 tasks in 43 languages. The benchmark's rankings closely align with established benchmarks, achieving a Spearman correlation of over 0.97 across 55 models, thereby enabling faster model selection and regression detection without the overhead of comprehensive evaluations.
HAKARI-Bench enables rapid, model-agnostic comparisons of retrieval architectures, achieving over 0.97 Spearman correlation with established benchmarks.
With the rapid spread of retrieval-augmented generation and semantic search, choosing the right embedding and retrieval configuration is increasingly hard. Large retrieval benchmarks are comprehensive but too heavy to rerun during development, and there is little infrastructure for comparing production settings--dimensionality reduction, quantization, reranking--across many models under identical conditions. We present HAKARI-Bench, a lightweight benchmark that reconstructs existing retrieval suites into small datasets (Nano-sets): 35 benchmarks and 551 tasks across 43 languages in a unified format, enabling same-condition, model-agnostic comparison of five retrieval families (BM25, dense, sparse, late interaction, rerankers) and their efficiency variants. Across 55 models, its overall ranking reproduces the official MTEB retrieval v2, MMTEB v2 retrieval, and English BEIR (full) at Spearman >0.97. HAKARI-Bench does not replace full evaluation; it enables rapid model selection, regression detection, and reading the quality-efficiency Pareto frontier. Code, data, and leaderboard are released under the MIT license.