Search papers, labs, and topics across Lattice.
This study examines the influence of different inference backends on the performance of instruction-tuned language models across various benchmarks and generation modes. By conducting a fully-crossed analysis involving three models, five backends, and six benchmarks, the authors reveal that backend choice can significantly affect model outputs, accounting for approximately 39% of the variability observed in benchmark scores. The findings highlight the necessity for transparency in reporting backend configurations and generation parameters to ensure accurate cross-backend comparisons.
Backend choice can distort benchmark scores by nearly 40%, challenging the assumption that model performance is solely a property of the model itself.
Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39\%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.
