Search papers, labs, and topics across Lattice.
This paper introduces MemeMind, a novel approach for offline context optimization that enhances agent performance by utilizing reference-guided trace construction to recover from failed rollouts. By employing TraceBuilder to gather necessary evidence and ToolGuide to summarize these traces into actionable instructions, MemeMind effectively addresses the challenges posed by complex meme interpretation tasks that require coordinated visual and textual understanding. The method demonstrates significant improvements over existing baselines, achieving up to 22% better performance on the MemeX benchmark, underscoring its efficacy in optimizing agent behavior in demanding contexts.
MemeMind recovers from failed rollouts by leveraging offline reference answers, leading to a remarkable 22% boost in performance on complex meme interpretation tasks.
Offline context optimization improves an agent by revising its instructions and examples while keeping the model frozen. This approach learns from rollouts on an adaptation set, but some queries produce only failed rollouts. In these cases, the optimizer sees no successful example of how the available tools can reach the correct answer. We introduce MemeMind, which uses an offline reference answer to recover this missing experience. TraceBuilder identifies the evidence required by the reference, executes text search, image retrieval, and visual grounding, and verifies the resulting tool trace before adding it to the adaptation buffer. ToolGuide then summarizes the collected traces into a shared guide and separate instructions for each tool. The reference answers and constructed traces are used only during adaptation, while inference uses the learned guides with a frozen model. We study this problem through Anime, Comic, and Game meme interpretation. These memes combine edited and ambiguous visual content, overlaid text, long tail franchise knowledge, and culture specific references. Their interpretation can require coordinated visual grounding, image retrieval, and text search, making them a demanding setting in which native rollout groups may fail together. We evaluate MemeMind on MemeX, a benchmark of 1,000 such memes annotated by experts. Across two Qwen3-VL models, two language partitions, and two independent judges, MemeMind improves over the strongest context optimization baseline by 22.0% and 21.1% on Qwen3-VL-30B-A3B, and by 8.1% and 8.0% on Qwen3-VL-235B-A22B under GPT-5 judging. Ablations and held out traces show that constructing successful tool use for failed groups provides the largest component gain and produces more effective evidence acquisition at inference time.
