Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper critically examines the concept of "accuracy" as it pertains to compliance with the EU AI Act, highlighting the divergent interpretations between the machine learning and legal communities. It identifies five key tensions—nature of accuracy, performance, scope of validity, ends, and statisticalness—that reveal conflicting expectations and misunderstandings surrounding accuracy. The authors argue that these frictions not only hinder effective compliance but also underscore the need for standardized measures and tools to enhance the validity of accuracy assessments in AI systems.
Accuracy in AI compliance is a battleground of conflicting interpretations, revealing deep fractures between legal and technical expectations that could impede progress.
The machine learning community progresses (in part) by improving the "accuracy" of its systems. The EU AI Act explicitly refers to "accuracy" as part of its compliance measures for high-risk AI systems. Are we talking about the same thing? This work presents "accuracy" as a case-study for differing requirements of social worlds, the technological machine learning community and the legal community. While competition on accuracy contributes to technological development, machine learning scholars simultaneously recognize accuracy's shortcomings regarding the usefulness and effectiveness of machine learning systems. The legal counterpart embraces the vagueness of "accuracy," leaving interpretative flexibility for technological and societal changes. At the same time, accuracy is a core element of compliance within the EU AI Act. We elaborate on five main tensions, (a) nature of accuracy, (b) notion of performance, (c) scope of validity, (d) ends, and (e) statisticalness, to show that the two communities project disparate, and sometimes contradictory, expectations on accuracy. Both legal and technical communities lack precise understanding of "accuracy" beyond the contextual boundaries of their community. The resulting frictions, \eg, based on the empirical or normative understanding of accuracy, are symptoms of an unresolved (and unresolvable) debate on what accuracy is. We constructively use the frictions to recommend baselines and interventional studies in standardization, and demand for tools to extend the validity of accuracy measurements.