Search papers, labs, and topics across Lattice.
FAB-Bench is introduced as an end-to-end framework for adaptive RAG benchmarking in the semiconductor manufacturing domain, addressing the challenges of domain complexity and costly expert assessments. It uses six diagnostic metrics to evaluate RAG systems, coupling retriever diagnostics with generator-level reasoning analysis across varying context windows. Experiments across multiple LLMs and RAG frameworks reveal distinct context-scaling behaviors and identify attention dilution as a key factor in performance degradation at longer context lengths.
RAG performance in semiconductor manufacturing doesn't always improve with larger context windows; FAB-Bench reveals logarithmic growth, early saturation, and cold-start dynamics across different LLMs and RAG frameworks.
Retrieval-Augmented Generation (RAG) has become critical for knowledge-intensive applications, yet evaluating its performance in vertical domains remains difficult due to domain complexity, diverse context scales, and heavy reliance on expert assessments that are costly, inconsistent, and non-scalable. We introduce FAB-Bench, an end-to-end framework for adaptive benchmarking of RAG systems in semiconductor manufacturing. FAB-Bench defines six diagnostic metrics measuring factual accuracy, contextual utilization, completeness, retrieval relevance, technical depth, and reasoning consistency. The framework couples retriever diagnostics with generator-level reasoning analysis across context windows of 4K-32K tokens, quantifying how retrieval precision and generative fidelity co-evolve as contextual scope expands. From over 1,300 generated candidates, we curated a high-quality benchmark of 200 query-answer pairs spanning three synthesis strategies: needle-in-haystack, intra-document multi-topic, and cross-document multi-hop. Systematic evaluation across four LLMs and four RAG frameworks reveals three distinct context-scaling behaviors: logarithmic growth, early saturation, and cold-start dynamics, and identifies attention dilution as the primary mechanism behind performance degradation at extreme context lengths. Cross-framework validation on three additional production RAG systems confirms evaluation portability.
