Search papers, labs, and topics across Lattice.
This study analyzes 14,727 security and privacy (S&P) prompts collected from a dataset of 3.2 million user-LLM conversations, categorizing them into nine distinct topics. The researchers found that while commercial LLMs, like GPT 5.5, provided satisfactory responses 98% of the time, they also exhibited inconsistencies, sometimes delivering contradictory information across multiple queries. This highlights a critical gap in LLM reliability for S&P inquiries, raising concerns about user trust and safety in real-world applications.
Commercial LLMs may seem reliable for security advice, but they can deliver contradictory responses, risking user safety and trust.
Large language models (LLMs) are widely used to fulfill users' information needs; users ask LLMs about the weather, pose educational questions, and consult them for legal assistance. One particularly understudied area is digital security and privacy (S&P), where users may seek LLMs' help on how to secure their online accounts or protect their computers from cyber attacks. To the best of our knowledge, no prior study has collected or analyzed the S&P questions users ask LLMs; prior research on LLM response quality relied on expert-authored S&P misconceptions or FAQs rather than user queries. Drawing from WildChat, a dataset of 3.2M user-LLM conversations collected in the wild, our study identifies 14,727 S&P prompts and categorizes them into nine categories covering a wide range of S&P topics. From the S&P prompts, we sampled 450 and performed a thematic analysis to characterize the S&P questions users ask LLMs. Separate from the thematic analysis, we curated 270 advice-seeking S&P prompts, where users ask for recommendations, guidance, or specific S&P information. We measured LLM response quality and consistency when posing the prompt to LLMs 10 times. We found that commercial LLMs outperform open-weight models (GPT 5.5 provided "good enough" responses on 98% of prompts; Llama 4 on 47%). However, among prompts that received high-quality responses on average, commercial models sometimes produce contradictory responses across runs, risking confusing or misleading users.
CMU ML