Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates how discrete prompt optimization, prompt tuning, and prefix tuning differentially impact internal computations in dense and MoE LLMs by analyzing how task information propagates through the model. The key finding is that these adaptation methods induce distinct computational pathways, with optimized text primarily affecting response construction, input-level prompts influencing content representations, and layer-wise prefixes maintaining direct attention control, further leading to divergent expert routing in MoE models. Crucially, common representation-level metrics fail to distinguish effective adaptations from placebos, highlighting the need to disentangle representation drift from task-aligned computational changes.
Prompt adaptation methods don't just change what LLMs say, they fundamentally rewire *how* they compute, with distinct mechanisms for text optimization, prompt tuning, and prefix tuning.
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.