Search papers, labs, and topics across Lattice.
The paper introduces MARCA, a bilingual (English and Portuguese) benchmark designed to evaluate LLMs' ability to perform web-based information seeking. MARCA uses 52 manually authored multi-entity questions with checklist-style rubrics to explicitly measure answer completeness and correctness. Experiments with 14 models reveal performance differences, the benefit of orchestration for improved coverage, and variability in cross-lingual transfer.
LLMs struggle to transfer web search abilities from English to Portuguese, highlighting the need for more robust multilingual evaluation.
Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA
