Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Relay-Bench is introduced as a comprehensive benchmark designed to evaluate LLMs' capabilities in solving complex, multi-domain reasoning tasks using a single prompt. The benchmark challenges models with composite problems that integrate various single-domain subproblems, requiring advanced reasoning and problem-solving skills across diverse domains such as coding, math, and general knowledge. The leading model, GPT-5.5 (xHigh), achieved a score of 43.3%, highlighting the current limitations of LLMs in handling intricate reasoning chains effectively.
LLMs struggle with multi-domain reasoning, as evidenced by GPT-5.5's mere 43.3% success rate on Relay-Bench's complex challenges.
Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs'ability to complete an assortment of tasks from distinct domains in a single prompt. The leading model, GPT-5.5 (xHigh), scores 43.3%. The test set entirely consists of composite problems: groups of single-domain subproblems that are strung together into challenges that require reasoning across multiple domains in combination. Many of these problems then have layers of complexity added through prompt encoding and deliberate context bloat. Domains tested include visual reasoning, coding, math, information extraction (with a focus on web search), problem-solving, general knowledge, and data analysis. No restrictions are imposed outside of the model harness, and models are explicitly encouraged to leverage code-execution, web searches, and all available tools. All problems are composed of two to thirteen subproblems and do not require multi-modal input or output.