Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors identify a systematic evaluation-time distribution shift in AlphaFold2-style architectures caused by the non-linear expectation gap $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))] \neq \mathrm{LayerNorm}(x)$, deriving a closed-form first-order correction (DLC) to resolve it. This mathematical discrepancy degrades inference when Dropout is sequenced directly before LayerNorm, an architectural pattern ubiquitous across biomolecular ML. Evaluated across nine protein folding models and one docking model, DLC matches the accuracy of expensive Monte Carlo dropout ensembling and yields consistent $0.3\%$ to $13\%$ performance gains with negligible computational overhead.
Chaining Dropout into LayerNorm silently degrades inference across structural biology models due to an uncorrected expectation gap, but a closed-form first-order adjustment recovers up to 13% accuracy at zero extra compute cost.
Although $\mathbb{E}[\mathrm{dropout}(x)] = x$, here we show that $\mathbb{E}[\mathrm{LayerNorm}(\mathrm{dropout}(x))]$ is not equal to $\mathrm{LayerNorm}(x)$. Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested ($\sim0.3\%-13\%$), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models. Code to reproduce the experiments is available at https://github.com/oxpig/DLC.