Search papers, labs, and topics across Lattice.
This paper introduces SkillSight, a novel retrieval framework designed to enhance skill selection in large language model agents by addressing the bias introduced by shared descriptive patterns in skill libraries. By employing Semantic Background Calibration and Lexical Evidence Calibration, SkillSight effectively reduces the influence of these common patterns, leading to significant improvements in retrieval accuracy. The framework achieves up to a 20.21 percentage point increase in Recall@10 and outperforms existing methods in both speed and overall performance across multiple agent models.
Shared descriptive patterns in skill libraries can obscure task-relevant signals, but SkillSight reveals and calibrates these biases to boost retrieval accuracy by over 20%.
As large language model agents gain access to increasingly large skill libraries, retrieving the right skill becomes critical to reliable capability selection and execution. Existing retrievers often treat skill descriptions as ordinary documents, overlooking their highly regular structure: shared descriptive patterns recur across many skills while providing little evidence for distinguishing the required capability. We show that this shared descriptive background systematically contributes to dense relevance scores, induces a pronounced energy gap between queries and skill documents, and obscures task-relevant signals. Based on this observation, we propose SkillSight, a training-free retrieval framework that calibrates shared background in both semantic and lexical spaces. Semantic Background Calibration estimates a background subspace from generic tokens identified by IDF, reducing similarity induced by shared descriptive patterns, while Lexical Evidence Calibration downweights shared background tokens to recover discriminative token-level evidence. Experiments on SRA-Bench and SkillBench-Supp demonstrate consistent improvements across retrieval metrics, with SkillSight improving Recall@10 by up to 20.21 percentage points over the original dense retriever. In end-to-end evaluation, SkillSight achieves the best overall performance across three agent models and outperforms LLM Selection by up to 4.97 percentage points. It is also up to 1,248 times faster than the Dense + Reranker baseline. These results identify shared descriptive background as a key source of bias in skill retrieval and demonstrate that explicitly calibrating it enables accurate and efficient skill selection without additional training. Our code is available at https://github.com/xiaojinying/SkillSight.
