Search papers, labs, and topics across Lattice.
This study evaluates the robustness of large language models (LLMs) in public health applications against two specific types of user input perturbations: misinformation framing (MF) and layperson rewriting (LR). The results indicate that MF significantly degrades model accuracy by an average of 7.2 percentage points, with prediction flip rates ranging from 9% to 38%, while LR only results in a modest 1.4 percentage point decrease. These findings underscore critical deployment risks, revealing that LLMs may produce erroneous outputs when faced with misinformation and misinterpret patient-reported symptoms expressed in everyday language.
LLMs lose up to 7.2% accuracy when faced with user-generated misinformation, revealing a hidden vulnerability in public health applications.
Large language models (LLMs) are increasingly applied in public health applications, yet their robustness to non-clinical user inputs remains underexplored. We propose a domain specific robustness benchmark that evaluates LLMs under two perturbation types that commonly arise when non-clinical users interact with health AI systems: misinformation framing (MF), where prompt might be injected by false health claims, and layperson rewriting (LR), where patients describe symptoms in everyday language rather than medical terminology. Our goal is to evaluate the stability of LLMs under these perturbation. Experiments show that MF degrades accuracy by 7.2 pp on average with prediction flip rates of 9-38 percent, even when claims are explicitly labelled as unsupported; LR causes only 1.4 pp degradation. These findings highlight two distinct deployment risks in public health settings: models may produce incorrect outputs when users unintentionally carry misinformation into their queries, and may misinterpret clinically relevant details when patients use informal language. Both risks call for perturbation-aware robustness evaluation beyond clean baseline benchmark
