Search papers, labs, and topics across Lattice.
This paper introduces a novel semantic pre-training framework that enhances the Tsetlin Machine (TM) by leveraging semantic clusters derived from pre-trained language models, bypassing the limitations of static word embeddings. By grouping text samples into semantically coherent clusters using K-means or Top2Vec, the TM learns interpretable keywords that improve its performance on downstream tasks. The proposed method significantly outperforms traditional TM approaches and achieves competitive results with BERT, all while maintaining interpretability, which is crucial for high-stakes applications.
Semantic clustering transforms the Tsetlin Machine into a powerful yet interpretable model, rivaling BERT's performance without sacrificing transparency.
Pre-trained language models such as BERT achieve strong text classification performance but lack transparency, limiting their use in high-stakes settings. The Tsetlin Machine (TM) offers fully interpretable, clause-based reasoning but captures little semantic information, and prior attempts to bridge the two rely on static word embeddings that miss contextual meaning. We propose a semantic pre-training framework that transfers knowledge from a pre-trained language model into a TM without using embeddings. Text samples are grouped into semantically coherent clusters with K-means or Top2Vec, and the resulting cluster-sample pairs pre-train a non-negated TM with enhanced Type I feedback. The TM thereby learns interpretable semantic keywords that are fine-tuned on downstream tasks. Across five datasets, our method substantially outperforms vanilla and embedding-based TMs and reaches performance competitive with BERT while remaining interpretable.
