Search papers, labs, and topics across Lattice.
This paper introduces Ko-WideSearch, a novel benchmark designed to evaluate the performance of web agents in exhaustive set enumeration tasks in Korean, addressing a significant gap in existing benchmarks that primarily focus on depth. The benchmark comprises 228 tables across 190 entities and is structured to test various difficulty levels by manipulating table width and composite keys, allowing for a systematic assessment of agents' abilities to enumerate complete sets and their attributes. Results reveal that while web agents excel in recovering set membership (Item-F1 of 92.8), they struggle with accurately filling out attribute rows (Row-F1 of 53.7), indicating a critical area for improvement in web agent capabilities.
Web agents can list items with high accuracy but falter dramatically when tasked with filling in detailed attributes, revealing a fundamental limitation in their breadth-search capabilities.
Web-agent benchmarks overwhelmingly measure depth -- pinning one obscure answer behind a chain of constraints -- while breadth, exhaustively enumerating a closed set and filling each item's attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce Ko-WideSearch, a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity -- a TV season, a dynasty, a league, an administrative region, an election -- and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently -- table width and a 2-D composite key -- so cross-product membership climbs from 0\% to 100\% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g.\ Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right.
