Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores a novel failure mode in transfer learning termed "loss shift," where the data distribution remains constant while the loss function changes, necessitating different representations of the same data. By employing Bayes quotients, the authors demonstrate that a representation optimized for a coarser loss is inadequate for a finer target loss, leading to an exact quantitative relationship between excess risk and the conditional information about the target variable that is lost. Experiments across various settings confirm that representations yielding equivalent classifications can exhibit divergent performance under different log-loss functions, highlighting the critical role of loss functions in representation learning.
Loss shift reveals that even with a fixed data distribution, the choice of loss function can fundamentally alter the required data representation, impacting model performance.
Transfer learning is usually studied as a consequence of distribution shift. This paper identifies an orthogonal failure mode in which the data distribution is fixed and the loss changes. This setting is called \emph{loss shift}. A loss determines which information in \(X\) is Bayes-relevant, and two losses may therefore require different representations even under the same joint law \(P(X,Y)\). The idea is formalized using Bayes quotients, which allow losses to be ordered by refinement. In the Bayes-quotient formulation, strict refinement gives an immediate qualitative obstruction. A source-minimal representation for a coarser loss is insufficient for a strictly finer target loss. For finite-output log loss, this obstruction becomes an exact quantitative identity. The excess risk is the conditional information about \(Y\) discarded by the representation. Experiments in controlled, learned, synthetic-image, and real-image settings show the predicted effect, i.e., classification-equivalent representations can have different optimal log-loss performance under a fixed data distribution.