Search papers, labs, and topics across Lattice.
This study introduces FiT, a diagnostic framework that evaluates small LLMs based on their capabilities in vocabulary recognition, parametric knowledge, and contextualization for cybersecurity QA. The empirical analysis of five 7-billion-parameter models reveals that fine-tuning can degrade essential knowledge components, with knowledge-focused tuning causing moderate degradation and instruction-focused tuning leading to significant knowledge loss. Importantly, pre-fine-tuning FiT scores can predict the impact of fine-tuning, allowing researchers to select models that minimize unnecessary adaptation costs and enhance deployment safety in cybersecurity applications.
Fine-tuning small LLMs for cybersecurity may actually degrade critical knowledge, with pre-fine-tuning diagnostics offering a way to avoid costly mistakes.
Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.
