Search papers, labs, and topics across Lattice.
RELISH is introduced as a parameter-efficient architecture for text regression with LLMs, directly predicting scalar values from frozen LLM representations. It iteratively refines a learned latent state via cross-attention over token-level representations, mapping the final state to a point estimate using a linear regressor. Experiments across five datasets and four LLM backbones demonstrate that RELISH consistently outperforms existing LLM regression methods while using significantly fewer trainable parameters (3.4-3.7M) compared to LoRA.
LLMs can now directly predict scalar values with a tiny, efficient head (3.4M parameters) that outperforms prior regression methods.
We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models. Rather than decoding numeric targets as text or aggregating multiple generated outputs, RELISH predicts scalar values directly from frozen LLM representations by iteratively refining a learned latent state through cross-attention over token-level representations, and then mapping the final state to a point estimate with a linear regressor. Across five datasets, four LLM backbones, and two LLM training regimes, RELISH consistently outperforms prior baselines from all three major LLM regression families, including autoregressive decoding, regression-aware inference, and existing predictive head methods. Despite these gains, RELISH remains highly parameter-efficient, requiring only 3.4-3.7M trainable parameters across frozen LLM backbones (only 0.01-0.04% additional overhead), far less than LoRA-based alternatives that grow with model size (0.26-0.42%).
