Search papers, labs, and topics across Lattice.
This paper introduces TOTEN, a knowledge-based ontological tokenization framework designed to enhance the representation of physical quantities and technical notation in Brazilian Portuguese by replacing traditional statistical methods with a formal ontology of engineering entities. By integrating deterministic coupling with external oracles, TOTEN achieves superior performance in ontological atomicity and numerical reconstruction compared to existing tokenization methods, demonstrating significant improvements in both internal and external evaluations. The results indicate that TOTEN not only preserves the integrity of technical expressions but also provides a robust framework for future applications in natural language processing within technical domains.
TOTEN outperforms state-of-the-art tokenization methods, achieving a remarkable 0.775-0.904 in numerical reconstruction accuracy, compared to just 0.627-0.703 for the best existing approach.
Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting physical quantities, numbers, units, and symbolic expressions into lexically arbitrary subwords. We present TOTEN, a knowledge-based ontological tokenization framework that replaces statistical derivation with declarative classification grounded in a formal ontology of engineering entities (OEE). We formalize TOTEN as the triple <O, classify, {inst_tau}>: the ontology gathers types, structural principles, composition relations, and preservable invariants; the classification function maps raw text into typed regions; and the instantiator family yields a self-descriptive structured representation. Robustness derives from deterministic coupling with three external oracles: Pint (dimensional), Unicode Character Database (typographic), and RSLP (Portuguese morphology). Intrinsic evaluation covers four properties verifiable by construction -- ontological atomicity, dimensional equivalence, typographic robustness, and numerical reconstruction -- over an internal, physically validated benchmark (EngQuant, N=800) and four Brazilian Portuguese external corpora (N=1771 eligible cases). We also report detection recall, distinguishing coverage from conditional atomicity. Against eight state-of-the-art baselines, TOTEN achieves unit ontological atomicity in all contrasts and numerical reconstruction of 0.775-0.904 on external corpora, vs. 0.627-0.703 for the best baseline (Quantulum3); on EngQuant, 0.780 vs. 0.340. Differences are statistically significant (McNemar with Holm correction). Spearman correlation between internal and external rankings confirms concurrent validity of the control benchmark. Dimensional equivalence shows statistical parity with Pint, the oracle from which the system inherits dimensional authority.
