Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This work investigates how different distribution shift evaluation strategies impact the perceived performance of drug-target affinity (DTA) models, moving beyond standard random splits. By curating a large dataset and comparing a baseline with 12 novel architectures across four distinct extrapolation scenarios (scaffold, fingerprint-cluster, protein-cluster, and dual OOD), the study reveals significant shifts in model rankings. Notably, model performance and relative superiority are highly sensitive to the type of distribution shift, with protein-cluster and dual OOD evaluations leading to substantially higher RMSE and reversed architectural preferences compared to chemical-focused shifts.
Drug-target affinity model performance rankings flip dramatically when evaluated on unseen protein targets, revealing a critical flaw in standard benchmarking.
Drug-target affinity (DTA) prediction is widely used to prioritize candidate compounds before costly experimental screening. DTA models are often compared under a single data split, even though deployment may require extrapolation to new chemical series, new protein targets, or both. We ask whether the distribution shift used for evaluation changes which architecture appears best. We curate 718,800 unique drug-protein pairs from the ChEMBL and BindingDB datasets. We compare a Morgan-fingerprint + protein-CNN baseline with 12 controlled architectures that combine four drug representations with three ESM-2 interaction modes. Mean validation RMSE increases from 0.950 and 0.945 under scaffold and fingerprint-cluster OOD to 1.299 and 1.321 under protein-cluster and dual OOD. Model rankings are similar across the two chemical shifts (tau = 0.79), but agreement with scaffold OOD falls under protein OOD (tau = 0.39) and reverses under dual OOD (tau = -0.55). Held-out evaluation, repeated seeds, group-aware bootstrap analysis, and a size-matched control support the same conclusion: architecture selection depends on the form of extrapolation, not only on average error or training-set size. DTA benchmarks should therefore match the chemical and target shifts expected at deployment.