Search papers, labs, and topics across Lattice.
This paper introduces torus embeddings as an alternative to Euclidean or hyperspherical embeddings, motivated by the prevalence of integer overflow arithmetic in computers. It demonstrates how to adapt common deep learning frameworks to create representations with toroidal topology, using both normalization-based and direct modulo strategies. The study finds that a normalization-based strategy yields stable training and performance comparable to hyperspherical L2 normalization, while also maintaining good quantization properties, suggesting a pathway to efficient TinyML implementations.
Torus embeddings offer a surprisingly simple route to efficient TinyML by natively matching the integer-with-overflow arithmetic of most hardware.
Many data representations are vectors of continuous values. In particular, deep learning embeddings are data-driven representations, typically either unconstrained in Euclidean space, or constrained to a hypersphere. These may also be translated into integer representations (quantised) for efficient large-scale use. However, the fundamental (and most efficient) numeric representation in the overwhelming majority of existing computers is integers with overflow -- and vectors of these integers do not correspond to either of these spaces, but instead to the topology of a (hyper)torus. This mismatch can lead to wasted representation capacity. Here we show that common deep learning frameworks can be adapted, quite simply, to create representations with inherent toroidal topology. We investigate two alternative strategies, demonstrating that a normalisation-based strategy leads to training with desirable stability and performance properties, comparable to a standard hyperspherical L2 normalisation. We also demonstrate that a torus embedding maintains desirable quantisation properties. The torus embedding does not outperform hypersphere embeddings in general, but is comparable, and opens the possibility to train deep embeddings which have an extremely simple pathway to efficient `TinyML' embedded implementation.
