Search papers, labs, and topics across Lattice.
This study investigates the reliability of generative AI in managerial decision-making within ambiguous business contexts, focusing on ambiguity detection, resolution, and sycophancy. A novel four-dimensional business ambiguity taxonomy was used in a human-in-the-loop experiment across strategic, tactical, and operational scenarios, with model outputs judged by an "LLM-as-a-judge" framework. The results show that while models can detect some ambiguities, they struggle with others, and ambiguity resolution improves response quality, while sycophancy varies by model architecture.
Generative AI's strategic advice in business is often unreliable due to undetected ambiguities and sycophantic tendencies, highlighting the need for human oversight despite its potential as a cognitive scaffold.
Generative artificial intelligence is increasingly being integrated into complex business workflows, fundamentally shifting the boundaries of managerial decision-making. However, the reliability of its strategic advice in ambiguous business contexts remains a critical knowledge gap. This study addresses this by comparing various models on ambiguity detection, evaluating how a systematic resolution process enhances response quality, and investigating their sycophantic behavior when presented with flawed directives. Using a novel four-dimensional business ambiguity taxonomy, we conducted a human-in-the-loop experiment across strategic, tactical, and operational scenarios. The resulting decisions were assessed with an "LLM-as-a-judge" framework on criteria including agreement, actionability, justification quality, and constraint adherence. Results reveal distinct performance capabilities. While models excel in detecting internal contradictions and contextual ambiguities, they struggle with structural linguistic nuances. Ambiguity resolution consistently increased response quality across all decision types, while sycophantic behavior analysis revealed distinct patterns depending on the model architecture. This study contributes to the bounded rationality literature by positioning GAI as a cognitive scaffold that can detect and resolve ambiguities managers might overlook, but whose own artificial limitations necessitate human management to ensure its reliability as a strategic partner.
