Search papers, labs, and topics across Lattice.
This paper introduces post-hoc conformal selection (PH-CS), a method that allows users to adaptively select test inputs based on a data-driven false discovery proportion (FDP) estimate, unlike existing conformal selection (CS) methods that fix the target FDR level beforehand. PH-CS constructs a path of candidate selection sets paired with FDP estimates, enabling users to optimize a utility function that balances selection size and FDR. The method leverages conformal e-variables and the e-Benjamini-Hochberg (e-BH) procedure, providing a finite-sample post-hoc reliability guarantee, ensuring the estimated FDP is a valid upper bound on the true FDR.
Now you can adaptively control false discovery rates in your selections *after* seeing the data, letting you balance selection size and FDR based on observed evidence and downstream needs.
Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR). Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data. For example, in genomics or neuroimaging, researchers often inspect the distribution of test statistics, and decide how aggressively to pursue candidates based on observed evidence strength and available follow-up resources. To address this limitation, we introduce {post-hoc CS} (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate. PH-CS lets the user select any operating point on this path by maximizing a user-specified utility, arbitrarily balancing selection size and FDR. Building on conformal e-variables and the e-Benjamini-Hochberg (e-BH) procedure, PH-CS is proved to provide a finite-sample post-hoc reliability guarantee whereby the ratio between estimated FDP level and true FDP is, on average, upper bounded by $1$, so that the average estimated FDP is, to first order, a valid upper bound on the true FDR. PH-CS is extended to control quality defined in terms of a general risk. Experiments on synthetic and real-world datasets demonstrate that, unlike CS, PH-CS can consistently satisfy user-imposed utility constraints while producing reliable FDP estimates and maintaining competitive FDR control.
