Search papers, labs, and topics across Lattice.
This paper introduces Tacet, a novel language and type system designed to enforce statistical validity in empirical comparisons within computer science research. By requiring analysts to declare their expectations and the outcomes they generate, Tacet ensures that claims made are statistically sound and prevents cherry-picking of results. The authors validate their approach through a machine-checked metatheory and practical case studies, demonstrating its effectiveness in real-world applications like the SWE-bench Verified leaderboard and BIG-Bench Hard.
Tacet ensures that every statistical claim in empirical research is backed by rigorous validity checks, eliminating the risk of cherry-picking results.
Empirical comparisons between systems are a standard form of evidence in computer science research, but few are checked for statistical validity: most are never framed as statistical tests at all. Existing multiple-comparison procedures could control the resulting error, but need inputs (what an analysis examined, and how its observations are arranged) that are not recoverable from a list of p-values. We introduce Tacet, a language in which an analysis declares what it generated, states what it expects to find, and is refused any claim it cannot afford or cannot properly test. Its core calculus T pairs a free estimation sublanguage, carrying a reported footprint and a purity bit that records whether any outcome was consulted in building a value, with a priced claim sublanguage, carrying a wealth transformer, connected only by a mechanism that prices a comparison. A sample selected by reading outcomes sets the purity bit and is recorded as having examined everything it read, permanently, so it can never be granted a one-sided or confirmatory price, without the system ever asking whether the analyst intended to cherry-pick. Whether a comparison is paired or clustered is computed statically from the artifact schema, from declared functional dependencies between key fields alone and before any data is read, and a mechanism that assumes that structure away is refused rather than priced. Because the wealth transformer is antitone in the realized p-value, affordability can be checked before the analysis runs too, turning pre-registration into a typing rule. We prove the metatheory machine-checked in Lean 4 with no admitted gaps, and demonstrate the approach on a reference implementation and two case studies on published artifacts, the SWE-bench Verified leaderboard and BIG-Bench Hard.
