Search papers, labs, and topics across Lattice.
StreamProfileBench is introduced as a benchmark for evaluating LLMs on fine-grained user profile inference in dynamic, real-world streaming scenarios using a dataset of 120,000 UGC posts from 7,000+ users. The benchmark formalizes user profiling as a continuous state maintenance task and uses an annotation-free evaluation framework based on temporal correlations of user interests. Experiments on 14 LLMs reveal a conservative bias, where models struggle to recognize interest decay and over-retain past interests, highlighting the challenge of continuous profile updating.
LLMs struggle to adapt to evolving user interests in streaming scenarios, exhibiting a conservative bias that over-emphasizes past interests and misses interest decay.
Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the reality of personalized systems, where User-Generated Content (UGC) arrives continuously and fine-grained profile evolve rapidly. To bridge this gap, we introduce StreamProfileBench, a large-scale benchmark for fine-grained streaming user profiling. We formalize streaming user profiling as a continuous state maintenance task and curate a highly authentic dataset comprising over 120,000 UGC posts from 7,000+ real users across five diverse platforms. By leveraging the temporal correlation of user interests, we further propose a novel, annotation-free evaluation framework. Extensive experiments across 14 leading LLMs reveal that continuous profile updating remains an open challenge. Models exhibit a systemic conservative bias, over-retaining past interests while failing to recognize interest decay. Ablation experiments further validate the practical utility and necessity of the streaming paradigm.
