Search papers, labs, and topics across Lattice.
The paper introduces PRISM-Δ, a novel prompt highlighting method that steers LLMs to prioritize specific text spans by decomposing the difference between positive and negative cross-covariance matrices to extract discriminative steering directions. PRISM-Δ uses a projection-based approach to maximize the energy of relevant contexts while eliminating shared structural patterns, and it assigns continuous importance weights to attention heads. Experiments across various benchmarks and models demonstrate that PRISM-Δ achieves state-of-the-art performance, with relative gains up to +10.6% and improved fluency compared to existing methods, while also scaling effectively to long-context retrieval tasks.
Achieve up to 10.6% performance gains in prompt highlighting while halving fluency costs by steering LLMs with PRISM-Δ, a method that decomposes cross-covariance differences to find discriminative directions.
Prompt highlighting steers a large language model to prioritize user-specified text spans during generation. A key challenge is extracting steering directions that capture the difference between relevant and irrelevant contexts, rather than shared structural patterns common to both. We propose PRISM-Δ (Projection-based Relevance-Informed Steering Method), which decomposes the difference between positive and negative cross-covariance matrices to maximize discriminative energy while eliminating shared directions. Each attention head receives a continuous softplus importance weight, letting weak-but-useful heads contribute at reduced strength. The framework extends naturally to Value representations, capturing content-channel signal that Key-only methods leave unused. Across four benchmarks and five models, PRISM-Δ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the fluency cost of steering. PRISM-Δ also scales to long-context retrieval, outperforming the best existing method by up to +4.8% relative gain. PRISM-Δ is compatible with FlashAttention and adds negligible memory overhead.
