Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This survey examines human-in-the-loop (HITL) methods for improving the safety and trustworthiness of NLP systems, focusing on auditing, robustness evaluation, data construction, and model steering. It highlights the necessity of human supervision to address limitations in bias detection, adversarial vulnerability, and generalization, particularly in low-resource settings and enterprise applications. The authors identify key gaps in scalable probing, robustness benchmarks, and governance of private systems, and propose research directions for adaptive auditing and collaborative evaluation.
Human-in-the-loop techniques aren't just about labeling data; they're now critical for validating model probes, verifying adversarial robustness, and governing private systems where automation falls short.
Large language models are widely deployed in high-stakes NLP tasks, yet risks such as bias, hallucination, adversarial vulnerability and unreliable generalization remain. Probe-based auditing reveals inconsistencies in model behavior. Adversarial text generation uncovers robustness gaps, especially in lower-resourced languages with limited benchmarks. Enterprise text-to-SQL settings expose the difficulty of validating outputs over private and large-scale databases. Human supervision is essential for probe validation, adversarial verification and domain-specific annotation, but it is costly and hard to scale. This survey examines recent human-in-the-loop methods that shift NLP from automation toward collaboration for safety and trustworthiness. We review how human expertise supports auditing, robustness evaluation, data construction and model steering. Our findings highlight gaps in scalable probing, sustainable robustness benchmarks, low-resource settings and governance of private systems. We outline practical research directions for adaptive auditing, collaborative evaluation and accountable deployment.