Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study explores the use of small language models (SLMs) for invoice categorization, emphasizing the importance of embedding geometry in financial contexts. By analyzing the pre-trained embedding space of a small sentence transformer (SBERT) and comparing it to a classic model (DeBERTa), the authors reveal that the embedding geometry is globally anisotropic yet contains locally isotropic clusters that correlate with vendor identity. Fine-tuning SBERT on a limited dataset achieves 0.96 accuracy in classification, outperforming both a zero-shot large language model and a vendor identity baseline, highlighting the effectiveness of SLMs for nuanced categorization tasks in finance.
Small language models can achieve high accuracy in invoice categorization by leveraging the geometry of sentence embeddings, even outperforming larger models in specific contexts.
Categorising invoices into the correct General Ledger (GL) code underpins financial reporting and tax compliance. This is a skilled accounting judgement rather than a routine task: the correct category depends subtly on the nature of the purchasing business, the vendor and the invoice text. Whilst AI is increasingly being adopted across industries to automate tasks, including invoice categorisation, implementations built on in-house small language models (SLMs) can simultaneously reduce cost and improve data security, confidentiality, and interpretability. We investigate this approach by first analysing the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM (DeBERTa). The sentence-embedding space of this financial corpus is globally anisotropic but composed of locally isotropic clusters, extending prior token-level findings to sentence embeddings in a financial setting, and these clusters are strongly correlated with the vendor identity. SBERT fine-tuned on a single GPU reaches 0.96 accuracy on invoice classification, above both a zero-shot LLM and a vendor identity baseline, increasing performance for smaller, challenging categories and new clients. For this important generalisation problem, SBERT reaches 0.9 F1 with roughly 100 client-specific invoices, showing that an in-house SLM implementation is promising. Combining these results with geometric analysis shows that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.