Search papers, labs, and topics across Lattice.
This paper introduces HelpBench, a benchmark designed to evaluate the ability of large language models (LLMs) to provide accurate advice on digital privacy, safety, and security. By curating 450 real-world questions and applying a rigorous evaluation rubric, the authors assessed responses from 18 state-of-the-art LLMs, revealing that while the average accuracy is commendable at 82%, approximately 10% of responses fall below a critical threshold of 65%, indicating potential harm. These findings underscore the necessity for improved reliability in LLMs to ensure they can be trusted as sources of assistance in sensitive digital contexts.
One in ten responses from leading LLMs on privacy and security advice could be dangerously misleading, highlighting a critical gap in their reliability.
This paper introduces HelpBench, a benchmark for assessing whether LLMs are capable of providing accurate help in response to questions about digital privacy, safety, and security. We curated 450 questions representing authentic user situations and developed rubrics for each question to evaluate the factual accuracy and tone of a response. Example questions touch on how to regain access to lost or suspended accounts, how to balance the trade-offs of hardware security keys versus other forms of two-factor authentication, whether a suspicious email is likely a scam, or whether an abuser might be able to track an individual based on their device peripherals. We then developed and applied an auto-rater to evaluate responses from 18 state-of-the-art LLMs. Our results indicate that while models provide high-quality advice (with scores of 82% on average), one in ten responses from models scores less than 65%, reflecting inaccurate and even harmful advice. Addressing these failures is critical for models to serve as trustworthy sources of assistance for digital privacy, safety, and security needs.
