Search papers, labs, and topics across Lattice.
This paper introduces the concept of contextual auditing as a method to evaluate AI systems, focusing on how these systems infer human behavior in motion capture scenarios. By contrasting actual system behavior with expected outputs, the authors reveal the importance of making implicit assumptions explicit and defining ground truth in the auditing process. The case study on skeleton inference highlights the utility of symmetry in auditing, particularly when ground truth is ambiguous or contested, paving the way for more robust evaluations of AI systems in practice.
Contextual auditing reveals hidden assumptions in AI evaluations, transforming how we assess systems when ground truth is elusive.
Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this question is AI auditing. Conducting an AI audit requires identifying how a system behaves (i.e., determining what types of inputs to audit it with and then observing and documenting actual system behavior) and contrasting that with how a system should behave (i.e., determining what the nominal outputs of a system should look like). We argue that this is best done through a contextual audit, which we introduce as a method for auditing measurements within the context of the practices that produce them. We show how contextual auditing enables the interrogation of assumptions implicit in the audit process and allows auditors to be explicit about what serves as ground truth, which we define as verifiable measurements about the real world against which systems are evaluated. We outline how the concept of symmetry from science and technology studies can enable audits when ground truth is unknown, unknowable, or contested. Finally, to demonstrate how contextual and symmetric audits can be conducted in practice, we present a case study of skeleton inference in motion capture and propose areas suggested by our case study as particularly fruitful for future audits of motion capture systems.
