Search papers, labs, and topics across Lattice.
This paper explores the optimization of on-premise retrieval-augmented factory agents by employing structural compression and retrieval-grounded adaptation to enhance their deployment on shop-floor hardware. The authors reveal that, contrary to traditional assumptions, model size does not predict retrieval-augmented answer quality, leading to a novel approach where sub-networks are selected based on performance metrics rather than size alone. Their findings demonstrate that a weight-shared supernetwork can effectively balance quality and throughput, achieving significant efficiency in a manufacturing context with minimal quality loss post-adaptation.
Model size is a poor predictor of answer quality in retrieval-augmented systems, challenging long-held assumptions in AI deployment.
On-premise assistants can give factory workers conversational access to machine documentation, but models capable of the task rarely fit shop-floor hardware. We show that after structural compression and retrieval-grounded adaptation, model size is no longer a reliable predictor of adapted answer quality: general capability falls almost linearly with parameter count, while judged retrieval-augmented answer quality does not. We therefore treat deployment as a post-adaptation selection problem, committing one sub-network per device on judged answer quality and measured on-device throughput under a configurable general-capability floor and memory budget; rules that optimize size, speed, or quality alone each give up capability or throughput. A weight-shared supernetwork trained with sandwich-style in-place distillation keeps this selection inexpensive. In a manufacturing-manual case study, extraction costs 13.7 percent of the unpruned model's judged quality and retrieval-grounded distillation returns it to within 4.6 percent, recovering two thirds of the loss, and the same assistant runs across three heterogeneous edge tiers at 1.3 to 5 watts standby.
