Search papers, labs, and topics across Lattice.
This study systematically compares training-free and training-based methods for intent classification in Large Language Models (LLMs), focusing on their accuracy, robustness, and failure modes. The findings reveal that while both approaches perform well on simpler benchmarks, training-based classifiers excel in more challenging scenarios, such as distinguishing between programming languages, whereas training-free methods demonstrate greater robustness against mixed-intent and adversarial prompts. This nuanced understanding of the trade-offs between these approaches can inform the design of more effective intent classification systems in LLMs.
Training-free methods outperform their training-based counterparts in robustness against adversarial prompts while still achieving competitive accuracy on simpler tasks.
Intent classification in Large Language Models (LLMs) involves categorizing user prompts into predefined classes. For instance, given a user prompt, the system must determine whether it primarily concerns mathematics, coding, or general text processing. Such classification enables routing prompts to specialized models optimized for specific domains, improving both accuracy and computational efficiency. In this work, we conduct a systematic study comparing training-free vs training-based approaches for intent classification. For this purpose, we consider two lightweight, training-free methods based on statistics of internal representations and compare them against MLP classifiers and linear probes. Our comprehensive empirical evaluation reveals that 1) Both training-free and training-based methods saturate easy benchmarks (mathematics vs. coding vs. natural language), 2) Training-based classifiers have an advantage on harder classification tasks (e.g. Java vs Python), and 3) Training-free methods are generally more robust to mixed-intent and adversarial prompts.
