Search papers, labs, and topics across Lattice.
This paper investigates the limitations of gradient-based task affinity estimation in multi-task learning (MTL), revealing that shared training instances are crucial for gradient conflicts to accurately reflect task relationships. They demonstrate a sharp phase transition in the effectiveness of gradient-based analysis, requiring at least 30-40% overlap in training data to reliably recover task relationships. By analyzing multiple datasets, the authors show that standard benchmarks often fall below this threshold, explaining the inconsistent results observed in MTL research.
Gradient-based task analysis in multi-task learning is fundamentally flawed on standard benchmarks due to insufficient overlap in training data, explaining years of inconsistent results.
Multi-task learning shows strikingly inconsistent results -- sometimes joint training helps substantially, sometimes it actively harms performance -- yet the field lacks a principled framework for predicting these outcomes. We identify a fundamental but unstated assumption underlying gradient-based task analysis: tasks must share training instances for gradient conflicts to reveal genuine relationships. When tasks are measured on the same inputs, gradient alignment reflects shared mechanistic structure; when measured on disjoint inputs, any apparent signal conflates task relationships with distributional shift. We discover this sample overlap requirement exhibits a sharp phase transition: below 30% overlap, gradient-task correlations are statistically indistinguishable from noise; above 40%, they reliably recover known biological structure. Comprehensive validation across multiple datasets achieves strong correlations and recovers biological pathway organization. Standard benchmarks systematically violate this requirement -- MoleculeNet operates at<5% overlap, TDC at 8-14% -- far below the threshold where gradient analysis becomes meaningful. This provides the first principled explanation for seven years of inconsistent MTL results.
