Search papers, labs, and topics across Lattice.
This paper introduces a black-box evaluation framework designed to assess the capability of Large Language Models (LLMs) in generating Design Structure Matrices (DSMs) from structured technical documentation. By benchmarking generated DSMs against manually validated ground-truth matrices, the framework reveals that while LLMs can produce structurally plausible DSMs with high reproducibility, they are significantly affected by ambiguities and inconsistencies in input. The findings underscore both the potential and limitations of LLM-driven DSM automation, providing a transparent benchmark for future Auto-DSM evaluations in model-based systems engineering (MBSE) workflows.
LLMs can generate structurally plausible Design Structure Matrices, but their performance is critically undermined by input ambiguity and inconsistent definitions.
This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs). The evaluation integrates both single-run and multi-run perspectives, combining structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss'$\kappa$). To synthesize these aspects, a Composite Quality Score (Q) is proposed. Controlled experiments are conducted on two datasets: a fictive abstract system and a real-world refrigerator decomposition, covering variations in phrasing, parameter-dataset alignment, and system complexity. Results show that LLMs can produce structurally plausible DSMs and achieve high reproducibility under well-structured inputs, but remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation. The findings highlight systematic sources of hallucination and abstention failure, demonstrating both the potential and current limitations of LLM-driven DSM automation. The proposed framework provides a transparent benchmark for auditing Auto-DSM pipelines and establishes foundations for integrating LLM-based decomposition methods into model-based systems engineering (MBSE) workflows.
