Search papers, labs, and topics across Lattice.
This paper introduces HERMES, a hierarchical labeling substrate that enhances data-mixing methods by allowing for a multi-granularity approach to document annotation. By employing a Learned Semantic Transform followed by a three-stage residual vector quantization, HERMES enables the annotation of documents into a flexible code structure that can adjust granularity dynamically. The key finding is that this approach improves task performance in a pre-training context, achieving a significant capability macro-average lift of +0.0253 on a 16-task benchmark by optimizing label granularity and coverage strategies.
A novel hierarchical labeling system reveals that flexible granularity can significantly enhance task performance in pre-training data mixtures.
Most data-mixing methods assume the corpus has already been partitioned into groups, and the choice of those groups determines what a mixer can express. Existing labels, including provenance, topic or format taxonomies, and flat embedding clusters, commit to one semantic axis at one granularity; changing the resolution rebuilds the labels. We argue the bottleneck is the label system, not the mixer, and provide a hierarchical one. HERMES is a data-derived labeling substrate: a Learned Semantic Transform followed by 3-stage residual vector quantization annotates each document once into a coarse-to-fine code whose prefix length controls granularity up to approximately 130k cells. At coarse granularity HERMES sits at a plateau with KMeans-family methods on standard clustering metrics, so the contribution is the substrate, not the clusterer. On 1B-parameter, 25B-token pre-training, the hierarchy exposes an interaction fixed-granularity pipelines cannot test: at one prefix length, a combined Stage-2 rule contrast, equal-subbucket coverage versus size-proportional within-bucket quality top-30%, lifts a 16-task capability macro-average by +0.0253; at the next finer level, the same rule loses its measurable edge as candidate pools contract approximately 5x. HERMES reframes data mixture design from choosing among fixed label sets to navigating a reusable, data-derived granularity hierarchy.
