Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Decoder-based Sense Knowledge Distillation (DSKD), a novel framework for incorporating structured lexical knowledge from sense dictionaries into decoder-style large language models. DSKD addresses the challenge of applying sense knowledge distillation to generative models, which are often overlooked in prior work focused on encoders. Experiments on various benchmarks demonstrate that DSKD significantly improves knowledge distillation performance, allowing decoders to learn structured semantics without sacrificing training efficiency.
Decoder-based Sense Knowledge Distillation (DSKD) lets generative models learn structured semantics without the inference-time overhead of dictionary lookups.
Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.