Search papers, labs, and topics across Lattice.
This paper introduces MetaInfer, an LLM-driven inference engine generator that allows users to specify only runtime constraints, resulting in the automatic creation of customized inference frameworks. By leveraging a multi-agent collaboration system and a contract knowledge base, MetaInfer reduces the complexity and maintenance costs associated with traditional inference frameworks while enhancing performance. Evaluation results demonstrate that MetaInfer effectively organizes generation constraints and validation feedback, leading to efficient and runnable inference solutions tailored to specific requirements.
Customized inference engines can now be generated automatically from user-defined runtime constraints, drastically simplifying the development process.
As LLM technology advances, the space of model families, compute hardware, quantization schemes, parallelization strategies, and specialized optimization kernels continues to expand, sharply increasing the code complexity and maintenance cost of general-purpose inference frameworks. Conventional software engineering uses multiple layers of abstraction to support diverse application scenarios, but these abstractions also increase system complexity and may introduce additional performance overhead. This paper presents metainfer, an'LLM-as-Compiler'approach in which users specify only the runtime constraints of an inference program. An LLM-driven multi-agent collaboration system, coupled with a contract knowledge base, then automatically generates a compact customized inference framework that satisfies these constraints. We evaluate metainfer from three perspectives: the effect of source-code reference, the runtime behavior and performance profile of engines generated under the zero-reference constraint on CKB-covered targets, and knowledge-base evolution for new model and platform scenarios. The results show that metainfer organizes generation constraints, validation feedback, and knowledge consolidation into a continuous closed loop, enabling runnable customized inference solutions to be generated from explicit knowledge. The code is publicly available at https://github.com/MetaInfer/MetaInfer.
