Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
To resolve severe metric illusions in single-cell perturbation modeling, the authors establish AmbiModBench, an evaluation framework that benchmarks models against split-matched training-mean baselines, gene-coordinate permutations, and embedding-space coverage metrics. Standard validation protocols routinely mask failure modes by rewarding models that merely predict shared cellular backgrounds or magnitude distributions that persist even under gene shuffling. Across K562, RPE1, and Norman datasets, the benchmark reveals that standard absolute gains largely reflect generic background shifts, whereas true target-specific generalization tracks representation-space proximity rather than aggregate training set size.
Most single-cell perturbation models only appear accurate because they memorize generic background responses—their benchmark scores remain high even when gene labels are completely shuffled.
Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasible. While computational models increasingly predict these responses, three evaluation deficiencies obscure what their scores demonstrate. First, absolute metrics cannot separate target-specific predictions from a shared background response. Second, common metrics remain high under gene shuffling, so gene-level accuracy is never verified. Third, a score at one training size says nothing about coverage, which depends on representation-space proximity and response-constraining power. We propose AmbiModBench, a specificity-aware, gene-resolved and coverage-aware benchmark. It pairs every score with a training-mean reference fitted on the same split, screens each readout by gene-coordinate permutation, and links embedding distance to response variation. Across K562, RPE1 and Norman, strong absolute scores largely reflect shared background rather than target-specific learning. Widely used readouts track response magnitude distributions rather than the affected genes. Detectable gain follows representation-space coverage rather than training-set size. Nonetheless, on RPE1 the protocol yields a reproducible target-specific gain across five additional splits and three gene selections, which absolute scores alone cannot distinguish from shared background.