Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a linear-time language identification classifier that utilizes compositional data analysis by modeling character and bigram frequency distributions as compositional vectors. By employing the centered log-ratio (CLR) transformation, the authors effectively map these distributions onto a zero-sum subspace, allowing for the use of Euclidean distances as Aitchison distances. The proposed method demonstrates robust accuracy across various text lengths, particularly excelling in longer sequences, highlighting its efficiency and interpretability compared to traditional neural and statistical approaches.
Compositional representations can achieve high accuracy in language identification while operating in linear time, challenging the dominance of resource-intensive neural methods.
Language identification is commonly addressed using either neural architectures or statistical n-gram models. Neural approaches typically require substantial computational resources, whereas classical frequency-based methods offer efficient linear-time performance, but rely on distance metrics that are not always appropriate for compositional data. This work models character and bigram frequency distributions as compositional vectors constrained to the simplex and mapped via the centered log-ratio (CLR) transformation bijectively onto the $(D-1)$-dimensional zero-sum subspace of $\mathbb{R}^D$, where Euclidean distances correspond to Aitchison distances. A pipeline is proposed, combining CLR-transformed unigram and bigram features with Laplace smoothing to address sparsity. The method is evaluated on six languages. Experimental results show that the proposed approach achieves robust accuracy across different text lengths, with strong performance for longer sequences. These findings indicate that compositional representations provide a deterministic and computationally efficient alternative for language identification, particularly in settings where interpretability and low resource consumption are essential.