Search papers, labs, and topics across Lattice.
This paper reveals that sequence-pooled normalization can provide global context for convolutional sequence labelers, effectively bypassing the limitations imposed by the model's receptive field. By analyzing the Jacobian of the normalization layer, the authors demonstrate that this approach can achieve near-optimal performance with significantly smaller receptive fields, particularly in tasks involving long runs of labels. Their findings indicate that using normalization can enhance model performance by an order of magnitude on both synthetic and real genomic data, while also complicating the attribution of contributions from model components.
Sequence-pooled normalization can deliver nearly the same performance as larger receptive fields, redefining our understanding of context in convolutional sequence labeling.
A convolutional sequence labeler's receptive field is routinely treated as the extent of the model's usable context: it sets dilation schedules, bounds streaming horizons, and underwrites locality claims. However, we show that this can be false: when a normalization layer computes statistics from the current input along the sequence at inference, those statistics open a sequence-spanning path that bypasses the convolutional receptive field to provide global context. We derive this from the layer's Jacobian (the criterion needs no experiment), and what the path carries has a closed form. On a synthetic labeling process with computable optima, the global summary that a sequence-spanning normalization encodes already supplies almost all of what a larger receptive field would buy where labels come in long runs: a network reaching 9 positions comes within 0.009 of the whole-sequence optimum, against a near-chance bound for its reach. Closing the path, by taking the same statistics per position, multiplies what enlarging the receptive field is worth by up to an order of magnitude on simulated genomes at every difficulty level tested and on real 1000 Genomes haplotypes. The same path also confounds attribution: ablating a trained network's receptive-field-enlarging blocks severs part of the path, overstating their contribution 8.3-16.1-fold relative to retraining from scratch. The substitution of normalization for receptive field fades as labels switch more often. Where labels run long, neither the receptive-field justification nor the ablation is wrong about its numbers, but both credit the wrong component.
