Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study introduces an automated pipeline leveraging large language models (LLMs) to generate structured decision scenarios with validated complexity, addressing the inefficiencies of manual scenario creation in cognitive decision-making research. The evaluation of 4,238 generated scenarios across various domains revealed nearly perfect agreement among five independent model families, with robust psychometric validation metrics demonstrating significant tier separation and a dominant complexity construct. Notably, while a speed-quality trade-off was observed, the system's strong psychometric properties enable reliable classification into complexity tiers, enhancing the infrastructure for cognitive assessments in AI systems.
Automated generation of decision scenarios achieves near-perfect psychometric validation, revolutionizing cognitive decision-making research.
Cognitive decision-making research depends on diverse scenarios with carefully controlled complexity, yet manual production is slow, inconsistent, and biased. We developed an automated pipeline that uses LLms to generate structured decision scenarios and validates their complexity through a composite framework rooted in established task-complexity theory. We evaluated 4,238 scenarios across multiple domains and complexity tiers. Measurement validation met rigorous psychometric standards. Agreement among five independent model families was nearly perfect, with an intraclass correlation coefficient of 0.997 and a kappa of 0.971. Known-groups validity demonstrated large separation between tiers, with an eta-squared of 0.587 and all pairwise comparisons significant at p less than .001. Factor analysis revealed a dominant complexity construct, with loadings between 0.87 and 0.96 across three frameworks, while interactivity formed a weaker secondary dimension at 0.34. Discriminant validity was limited by a strong relationship between complexity and text length that persisted after controlling for tier, yielding a partial correlation of 0.86. This constrains construct purity but does not undermine the instrument's tier-grading function. Model analyses showed a negative association between throughput and schema pass rate (r = -0.967, p = .007, n = 5), suggesting a speed-quality trade-off, though largely driven by one high-throughput model. Llama 4 Maverick generated scenarios fastest at 134 per minute versus 25 for DeepSeek Chat V3.2, but underproduced complex-tier scenarios, whereas DeepSeek Chat V3.2 balanced domain coverage with high schema compliance. The system demonstrated strong psychometric properties, enabling reliable classification into Simple, Moderate, and Complex tiers and providing the measurement infrastructure needed for downstream cognitive assessment of AI systems