Search papers, labs, and topics across Lattice.
This paper introduces HySAT (Hyperbolic Structure-Aware Training), a novel training approach that applies hyperbolic geometry exclusively at the loss layer, addressing the limitations of traditional Euclidean transformers in expert domains structured as trees. By conducting extensive experiments across six expert SLMs, including Llama 3.1 and EXAONE 3.5, the authors demonstrate that this loss-only placement leads to stable training outcomes, in contrast to conventional methods that struggle with training collapses. The results reveal that hyperbolic geometry can effectively preserve the manifold structure of expert domains, enabling robust model performance without the typical training failures associated with adapter strategies.
Hyperbolic geometry at the loss layer can stabilize training in expert AI models, overcoming the pitfalls of traditional methods that dilute structural integrity.
Expert domains are trees; the Euclidean transformer is not, diluting parent-child structure exponentially at depth. The hyperbolic turn left one question unasked: not how much of a network to curve, but where curvature may touch the gradient. Placement is a law, not a knob: the same geometry on a trainable adapter collapses training (seventeen training collapses, ~220 GPU-hours), yet at the loss layer alone it trains without one -- this is HySAT (Hyperbolic Structure-Aware Training), hyperbolic losses at the loss layer only. Across six expert SLMs we constructed and deployed (Llama 3.1 and EXAONE 3.5; four adapter strategies; 18.0M-sample corpus; zero NaN over ~317K optimizer steps), a matched four-arm ablation isolates the preserved manifold invariant, and three propositions and a lemma prove why loss-only placement is stable where adapter-on-manifold is not. Four models are operationally deployed (one live, consumer-facing), two open-weight, with per-step traces and a seventeen-incident failure ledger on Zenodo (CC-BY-4.0).
