Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of large language models (LLMs) as rerankers in cold-start recommendation systems across five domains, revealing that calibrated LLM rerankers often underperform compared to traditional collaborative and content-based methods. In scenarios where the gold item is not guaranteed present, retrieval bottlenecks become pronounced, with standard retrievers identifying the gold item only 4.6-22.9% of the time due to a high proportion of new items lacking training interactions. The introduction of a learned hybrid fusion layer (LHF) significantly improves retrieval performance, outperforming all single retrievers and recovering substantial coverage in content-rich domains, although the gains are limited in collaboratively strong domains.
LLMs may struggle in cold-start scenarios, but a novel learned hybrid fusion layer can recover significant retrieval performance, outperforming traditional methods across multiple domains.
Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes. We test this assumption with a five-domain benchmark that explicitly separates reranking quality from retrieval coverage. In a positive-controlled regime where the gold item is guaranteed present, calibrated LLM rerankers fail to consistently outperform strong collaborative and content baselines under natural traffic, and within-family scaling from Qwen3-8B to Qwen3-32B narrows but does not close the gap on most domains. In a retrieval-realistic regime where the gold item is not injected, the bottleneck is more severe: standard single retrievers place the gold item in a 200-item pool only 4.6-22.9% of the time, largely because 32-91% of cold-start targets are brand-new items with no training interactions. We introduce LHF, a validation-trained learned hybrid fusion layer over a multi-retriever union pool, as a retrieval-side realizability baseline. LHF is the only combiner we test that beats every single retriever on all five domains and recovers 17-61% of oracle coverage headroom on content-rich domains, but only 5-7% on collaboratively strong domains. End-to-end experiments reveal the remaining mismatch: learned non-LLM ranking exploits the LHF pool, while prompt-level LLM reranking often degrades it. LLMs exhibit pockets of semantic cold-start advantage, especially in text-rich domains when the item is already present, but this advantage is largely unreachable in current retrieve-then-rerank pipelines. We release the benchmark protocol, splits, prompts, evaluation tooling, and archived reproducibility artifacts: data at https://doi.org/10.5281/zenodo.20991039 and code at https://doi.org/10.5281/zenodo.20993306.
Stanford HAI