Search papers, labs, and topics across Lattice.
This paper introduces the Stance Inference Complexity Index (SICI), a seven-dimensional measure that quantifies the semantic-pragmatic complexity of stance detection tasks for prompt-based LLMs. The authors demonstrate that SICI effectively predicts LLM accuracy across multiple datasets, revealing a phase-transition-like behavior in LLM errors as complexity increases. Notably, while stronger models shift the boundaries of error regimes, common interventions like prompting and retrieval do not eliminate high-complexity challenges but rather alter the models' attribution tendencies.
LLMs exhibit a surprising phase transition in error patterns as semantic complexity increases, challenging conventional approaches to stance detection.
Prompt-based LLMs are increasingly used for stance detection, but harder examples are not always repaired by clearer instructions, reasoning prompts, retrieval, or debate. We introduce SICI (Stance Inference Complexity Index), a seven-dimensional diagnostic measure of the semantic-pragmatic burden imposed by a target--text pair. Across SemEval-2016 and VAST, SICI predicts LLM accuracy better than surface proxies and shows substantial cross-scorer reliability ($\alpha=0.771$). More importantly, LLM errors change regime as SICI increases: low-complexity examples invite over-attribution, especially Against predictions; intermediate examples form an unstable boundary; and high-complexity examples rapidly concentrate on None. This phase-transition-like structure persists across GPT-3.5, GPT-4o-mini, DeepSeek-V3, and GPT-4o, although stronger models move the boundaries. A 15-method intervention study further shows that prompting, retrieval, and debate often shift models along the attribution--abstention axis rather than removing the high-complexity bottleneck.
