Search papers, labs, and topics across Lattice.
To eliminate the parameter bottleneck where sub-million-parameter language models spend a third of their capacity on the output projection matrix, this work develops Riemannian Language Models (RiLM), which decode next-token distributions directly from squared geodesic distances between a manifold-bound hidden trajectory and input embeddings. Addressing boundary collapse in naive hyperbolic recurrence through M枚bius stabilization, the authors evaluate flat Euclidean and Poincar茅 ball implementations against standard recurrent and attention baselines. At ~290k parameters ($d=128$), the hyperbolic model (HypRiLM) achieves 54.2 validation perplexity on WikiText-2, outperforming parameter-matched tied Transformers, LSTMs, and state-space models by more than 2x.
Eliminating the language model output projection matrix in favor of geodesic decoding on a Poincar茅 ball cuts perplexity by over 50% compared to tied Transformers and SSMs at sub-million parameter scale.
Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transformer at embedding width d = 128 still spends roughly one third of its capacity on the output matrix W_out in R^(d x |V|). We propose Riemannian Language Models (RiLM), which remove that layer entirely: context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings. The same embedding map serves input and output -- decoding is geometry. We instantiate the framework on flat R^d (Flat RiLM) and the Poincare ball H^d (HypRiLM) with a shared MLP composition map phi (~290k parameters, d = 128, |V| = 2000). Across five seeds on WikiText-2, HypRiLM reaches 54.2 +/- 0.2 validation perplexity versus 87.6 +/- 0.6 for Flat RiLM; tied and matched LSTM, Transformer, and SSM controls remain at 113-147 PPL on WT-2 -- HypRiLM leads by roughly 2x over the strongest tied recurrent baseline (SSM, 113.0 +/- 3.8). Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V|, while hyperbolic curvature helps selectively. We also characterize boundary collapse in naive hyperbolic recurrence and show how Mobius stabilization restores trainability. Claims are scoped to controlled small-model comparisons, not full-vocabulary state of the art.
