Search papers, labs, and topics across Lattice.
This paper investigates the enhancement of local LLM-based SSH honeypots through strategic prompt design and supervised fine-tuning, addressing the limitations of cloud models such as instability and cost. By evaluating eight models, including a fine-tuned GPT-3.5 and seven open-weight local models, the study reveals that while prompt structure significantly impacts shell emulation accuracy, fine-tuning effectiveness is contingent on dataset coverage. Notably, expanding the fine-tuning dataset with honeypot logs leads to substantial improvements in model performance, although the interaction between prompting and fine-tuning can create conflicting outcomes in shell behavior.
Local LLM-based SSH honeypots can achieve superior shell emulation accuracy with the right prompting and fine-tuning strategies, but their effects can conflict in unexpected ways.
LLM-based SSH honeypots often use closed cloud LLMs because they give strong shell realism, but cloud models create deployment problems. These include no stable versioning, provider-side changes, attacker-driven cost, and model decommissioning. Local open-weight models avoid these problems, but they usually perform worse and make mistakes that reveal the honeypot. These mistakes include malformed outputs, command echoing, inconsistent filesystem state, and AI-style artifacts. This paper studies how to improve and evaluate the shell emulation accuracy of local LLM-based SSH honeypots using prompt design and supervised fine-tuning. We fine-tune and evaluate eight models in total: the original fine-tuned GPT-3.5 model used in shelLM and seven open-weight local models, each compared to its base model. We also test how prompt structure transfers across model families. Using 34 automated unit tests that measure shell emulation accuracy in single-session and fresh-session settings, we find that prompt design has a large effect and that fine-tuning depends on dataset coverage. Fine-tuning on the original 112-conversation dataset does not improve aggregate pass rate, while an expanded dataset built from honeypot logs produces clearly stronger local models. Taken together, the results suggest that prompting and fine-tuning can each improve local LLM honeypots on their own, but their effects do not combine straightforwardly, since strong rule-based prompting and supervised adaptation can also conflict by addressing overlapping shell-behavior constraints.
