Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces the Relational Task Extrapolator (RTE) to improve generalization to unseen tasks outside the training distribution. RTE decomposes target tasks into known anchor tasks and learned transformations, enabling the model to extrapolate by applying learned relationships between tasks. Experiments across function and sequence prediction tasks demonstrate that RTE significantly outperforms existing methods in parameter, length, and compositional extrapolation.
Extrapolation isn't about memorizing tasks, it's about learning the relationships *between* them, and this relational approach unlocks surprisingly strong generalization.
Modern learning systems excel at interpolation but struggle to generalize to unseen tasks outside the training distribution's support. This failure occurs even in simple settings, such as handling task parameters beyond the training range, and persists despite advances in foundation models. To this end, we develop the Relational Task Extrapolator (RTE), an algorithm designed to enable systematic extrapolation to novel tasks. The key observation is that extrapolation is inherently relational: extrapolating to unseen tasks requires learning how tasks transform into one another. If a model learns the transformation between tasks A and B during training, it can apply that same transformation to relate known tasks to unseen ones at test time. RTE operationalizes this idea by decomposing each target task into a known anchor task and a transformation linking the anchor and target. It then learns a relational operator, mapping an anchor-transformation pair to predictions for the target task. We instantiate RTE across multiple task extrapolation regimes in function prediction, e.g. where target tasks use out-of-range parameters (parameter extrapolation), have greater compositional depth (length extrapolation), and/or recombine function primitives in unseen ways (compositional extrapolation). We further extend RTE to sequence prediction, integrating it into fine-tuning algorithms for foundation models. Across empirical studies, we find that RTE substantially outperforms existing approaches on extrapolation to novel, unseen tasks.