Search papers, labs, and topics across Lattice.
This paper addresses the limitations of existing methods for aligning Large Language Models (LLMs) that rely on explicit human feedback by introducing a novel approach that leverages implicit feedback through mouse and eye-tracking data. By creating the IFLLM dataset, which captures diverse user interactions with LLM responses, the authors demonstrate that incorporating this implicit feedback significantly enhances the performance of traditional text-based reward models. The results show an improvement in accuracy from 55% to 64% and a nearly threefold increase in response quality across eight LLMs, highlighting the importance of implicit feedback in model alignment.
Implicit feedback from user interactions can boost LLM alignment accuracy by nearly 20%, revealing a hidden layer of user preference that explicit feedback misses.
To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text. These existing methods have two key limitations. First, the users rarely provide explicit feedback for LLM responses, which makes the high-quality preference annotation expensive to collect. Second, the methods do not leverage implicit human feedback, which has proven vital to the economic moats of Internet giants. To quantify the value of implicit feedback, we build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers, their mouse trajectories, and eye gazing points to the LLMs' responses from their webcams. IFLLM shows that the users have very diverse types of gazing behavior and mouse trajectories. Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements after applying the DPO to eight LLMs, demonstrating the value of implicit feedback in the wild. Our data collection website, dataset, and codes can be found at https://github.com/themehulpatwari/llm-implicit-feedback/.
