Search papers, labs, and topics across Lattice.
We investigate whether pretrained tabular foundation models (TFMs) adhere to a fixed-weight in-context learning paradigm, analogous to linear regression and kernel smoothing, by developing two certificates designed to bypass the standardization trap inherent in public TFM packages. These certificates, which only rely on predictions at standardized labels, can reject both fixed-weight prediction and sums of independent nonlinear label transformations. Our evaluation across five TFMs reveals evidence of joint label processing, where altering one context label impacts the influence of others, a phenomenon that emerges during training and is largely reflected in attention scores.
Tabular foundation models exhibit joint label processing, deviating from fixed-weight in-context learning and suggesting context labels dynamically alter each other's influence on predictions.
Linear regression and kernel smoothing offer tractable explanations of in-context learning: in both, the features determine the weight assigned to each context label. However, whether this fixed-weight account describes pretrained tabular foundation models (TFMs) remains unclear. Testing this account using derivatives runs into a standardization trap: public TFM packages standardize the labels before the model sees them, yet ordinary derivatives also reflect behavior outside the set of standardized labels, making a model appear nonlinear even when every prediction it makes agrees with a fixed-weight map. We propose two certificates that depend only on predictions at standardized labels and can reject two distinct explanations: fixed-weight prediction and sums of independent nonlinear label transformations. Across the five public TFMs that we evaluate, our certificates show that changing one context label alters how other labels influence the prediction, a behavior we call joint processing. We further find that joint processing emerges with training and that attention scores carry most of the measured interaction. Together, these findings motivate TFM explanations that account for how context labels change the influence of individual examples.
