Search papers, labs, and topics across Lattice.
The paper introduces Object Aligner (OA), an open-source Python library designed to score the similarity of JSON objects by recursively aligning their structures, addressing the challenges of measuring output fidelity in large language models (LLMs). By employing the Hungarian algorithm and sequence alignment techniques, OA provides a deterministic scoring mechanism that adapts to various tasks through JSON Schema extensions, rather than requiring code modifications. A key innovation, referential alignment, allows OA to handle complex graph structures and generate ranked repair suggestions, demonstrating its effectiveness as a reward mechanism in the GEPA prompt optimizer across diverse datasets.
Object Aligner achieves robust JSON similarity scoring by inferring identifier bijections, enabling accurate evaluation of complex structured outputs without the pitfalls of traditional methods.
Large language models (LLMs) are often asked to produce JSON conforming to a fixed schema, powering information extraction, tool calling, agentic planning, and knowledge-graph construction. Measuring how closely an output matches a gold reference is essential yet surprisingly hard: exact match is brittle, text similarity ignores structure, and an LLM judge is expensive, opaque, and non-deterministic. We address this with Object Aligner (OA), an open-source Python library that scores two JSON objects deterministically by recursively aligning their trees (the Hungarian algorithm for unordered collections, sequence alignment for ordered ones) and awarding partial credit at the granularity the schema declares. The Object Aligner is configured entirely through a set of JSON Schema extensions, so adapting it to a new task involves annotating a schema rather than writing code. Complex structured data, however, are rarely flat trees: records may form graphs or hypergraphs keyed by arbitrary identifiers, breaking the assumptions of prior similarity metrics. Our central contribution, referential alignment, closes this gap by inferring a bijection between gold and candidate identifiers and scoring every reference through it, so the score is invariant to relabeling. Since recovering this bijection exactly is graph isomorphism, the Object Aligner approximates it with Weisfeiler-Leman color refinement. An order-sensitive sequence regime targets ranking and planning. Since the same alignment localizes every mismatch, the Object Aligner emits ranked repair suggestions at no extra cost. Used as a reward inside the GEPA prompt optimizer, Object Aligner helps or stays neutral across all datasets.
