Search papers, labs, and topics across Lattice.
This paper investigates the sensitivity of instruction-tuned embedding models to prompt phrasing across 11 datasets and 6 models, using 15 task-specific prompts per dataset. They find that single-prompt evaluations misrepresent the true distribution of scores and can either over- or under-estimate model performance. Critically, the study demonstrates that leaderboard rankings are unstable, as any model can be made to appear superior through favorable prompt selection.
Leaderboard rankings for instruction-tuned embedding models are meaningless, as prompt selection alone can crown any model the winner.
Instruction embedding models have become common among state-of-the-art models, however are evaluated using a single prompt per task. The single-point evaluation ignores a main problem of the instruction-based approach namely: sensitivity to the phrasing of the instruction. We present an empirical study of prompt sensitivity across 6 embedding models, 11 datasets, and 15 task-specific prompts per dataset, a total of 990. We show that reported scores misrepresent the distribution of scores over plausible prompts. The default prompt can both systematically understate or overstate performance. Furthermore, we show that the leaderboard ranking is not robust to prompt selection: by choosing prompts favorably, any model in our study can be promoted to first place. Our findings suggest that single-prompt evaluation is insufficient for instruction-tuned embedding models and that benchmarks should incorporate prompt robustness, either by evaluating over multiple prompts or by reporting sensitivity alongside point estimates.
