Search papers, labs, and topics across Lattice.
This paper introduces Kaleidoscope, an integrated evaluation workflow designed to address the challenges of aligning AI applications with local policy and governance requirements. By combining persona-based test generation, contextualized rubrics, and human review, Kaleidoscope enables reliable automated scoring that is both inspectable and iterative. Early pilot results demonstrate its effectiveness across diverse use cases, showcasing features that enhance the reliability of automated evaluations in real-world contexts.
Kaleidoscope reveals that a structured, context-aware evaluation process can significantly enhance the reliability of automated scoring in AI applications.
Evaluations (Evals) are a deployment bottleneck for real-world AI applications: public benchmarks rarely match a team's users, context, or policies, and human review is often tedious to scale. Motivated by our work with AI applications in the public sector, this project addresses recurring evaluation challenges encountered when applications must satisfy local policy and governance requirements. We present Kaleidoscope, an integrated workflow for contextual functional evaluation that links persona-based test generation, contextualized rubrics, and human review for reliability-gated automated scoring. Generated test cases are scored against application-specific rubrics; human annotations provide reviewable labels; and LLM judges automate scoring only when their agreement with those labels meets a configured threshold. Kaleidoscope is therefore a practical, inspectable, iterative workflow for product teams. We report early evidence from a three-week pilot across four organizational use cases and custom-rubric judge experiments on 108 annotated Q\&A pairs spanning four domains and 14 evaluation dimensions. The results highlight useful features for end-to-end reliable, automated scoring.
NUS