Search papers, labs, and topics across Lattice.
This paper benchmarks the performance of cloud-based and locally-hosted large language models (LLMs) on two system dynamics tasks: causal loop diagram (CLD) extraction and interactive model discussion. Cloud models excel at CLD extraction, while local models achieve competitive performance but struggle with long-context prompts in interactive discussions, particularly error fixing. The study systematically analyzes the impact of model type, backend (GGUF vs. MLX), and quantization levels, revealing that backend choice significantly affects performance due to differences in JSON schema enforcement and long-context handling.
Forget quantization levels – your choice of backend (GGUF vs. MLX) is the real bottleneck when deploying LLMs for system dynamics tasks, especially when JSON schema constraints and long contexts come into play.
We present a systematic evaluation of large language model families -- spanning both proprietary cloud APIs and locally-hosted open-source models -- on two purpose-built benchmarks for System Dynamics AI assistance: the \textbf{CLD Leaderboard} (53 tests, structured causal loop diagram extraction) and the \textbf{Discussion Leaderboard} (interactive model discussion, feedback explanation, and model building coaching). On CLD extraction, cloud models achieve 77--89\% overall pass rates; the best local model reaches 77\% (Kimi~K2.5~GGUF~Q3, zero-shot engine), matching mid-tier cloud performance. On Discussion, the best local models achieve 50--100\% on model building steps and 47--75\% on feedback explanation, but only 0--50\% on error fixing -- a category dominated by long-context prompts that expose memory limits in local deployments. A central contribution of this paper is a systematic analysis of \textit{model type effects} on performance: we compare reasoning vs.\ instruction-tuned architectures, GGUF (llama.cpp) vs.\ MLX (mlx\_lm) backends, and quantization levels (Q3 / Q4\_K\_M / MLX-3bit / MLX-4bit / MLX-6bit) across the same underlying model families. We find that backend choice has larger practical impact than quantization level: mlx\_lm does not enforce JSON schema constraints, requiring explicit prompt-level JSON instructions, while llama.cpp grammar-constrained sampling handles JSON reliably but causes indefinite generation on long-context prompts for dense models. We document the full parameter sweep ($t$, $p$, $k$) for all local models, cleaned timing data (stuck requests excluded), and a practitioner guide for running 671B--123B parameter models on Apple~Silicon.
