Search papers, labs, and topics across Lattice.
This paper introduces FisherSketch, a novel method for training-free source selection in large language models (LLMs) that share vocabularies but differ in prediction targets, particularly in scientific domains. By revealing that head Fisher alignment can be efficiently estimated as a cosine between kernel mean embeddings in the joint activation-error space, the authors address the limitations of traditional representation metrics and costly update-geometry computations. The results demonstrate that FisherSketch allows for practical head Fisher alignment estimation, enabling a deeper understanding of task similarity in LLMs even when activation similarity fails to provide insights.
Head Fisher alignment can be efficiently estimated in LLMs, revealing critical insights into task similarity that traditional metrics miss.
Training-free source selection for LLM families with shared vocabularies arises in scientific string domains such as SMILES, protein, and genomic sequences, where candidate corpora share a tokenizer but differ in prediction targets. This creates an activation-dark regime: representation-similarity metrics can be uninformative without assumptions about label-conditioned error geometry, while classical update-geometry metrics are computationally prohibitive at vocabulary scale. We show that, in a shared-output head setting, representation metrics (e.g., CKA) are non-identifiable for transfer; models can share identical representations yet have orthogonal head updates. The key identity is that head Fisher alignment is exactly a cosine between kernel mean embeddings in the joint activation-error space, exposing activation, error, and coupling factors rather than requiring a materialized Fisher matrix. FisherSketch estimates this cosine directly in a single streaming pass, making K=128,256 head Fisher alignment practical with a 16 KB task signature (m=4096) and a 192 KB per-task streaming state, small enough to store next to a model hash, but encoding transfer-relevant update structure. Beyond source selection, the same signatures and marginals provide a diagnostic instrument for studying whether LLM task similarity is driven by activations, errors, or their coupling; shared-parameter and internal-layer validations, together with Llama-3.1-8B verbalizer-shift experiments, show that FisherSketch remains informative when activation similarity cannot distinguish tasks.
