Search papers, labs, and topics across Lattice.
This paper introduces Graph-Structured Rubrics (GSR), a novel framework that transforms traditional rubric-based evaluations into structured, typed evaluation graphs, enhancing the clarity and precision of judgment criteria. By employing a systematic approach to compile rubrics into graphs that explicitly define criterion composition, GSR enables more accurate scoring and preference assessments for language model outputs. The results show that GSR significantly improves score agreement and pairwise accuracy compared to existing methods, demonstrating its effectiveness in structured evaluation contexts.
GSR boosts scoring accuracy for LLM evaluations by up to 6.75 percentage points, redefining how we structure and interpret rubric-based assessments.
Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
