Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces AQuAP, a dashboard designed for monitoring the health and quality of item banks in AI-driven assessment systems, addressing the complexities of large-scale educational assessments. By defining metrics such as Effective Bank Size (EBS) and various bank-health indicators, AQuAP provides actionable insights into item utilization, security, and diversity, ensuring the integrity of high-stakes tests. The application of AQuAP is exemplified through its integration with the Duolingo English Test, demonstrating its practical relevance in real-world testing scenarios.
Effective Bank Size (EBS) reveals how many unique test sessions can be created before content repetition, transforming item bank management in high-stakes assessments.
The large-scale digitization of educational assessment has made the continuous oversight of item banks both essential and complex. This paper presents Analytics for Quality Assurance for Item Pools (AQuAP), a dashboard environment for monitoring item quality and item bank health. AQuAP supports the operational implementation of the large scale item generation procedures for high-stakes tests as included in the Item Factory, a framework for automated and human-supported test development. The paper describes AQuAP in relationship with the process of item development, outlines the broader metric framework for item-pool quality assurance, and highlights the Effective Bank Size (EBS) as one central indicator of pool vitality. EBS quantifies how many independent test sessions can be constructed before content repetition occurs and, when coupled with exposure and usage metrics, provides insight into item bank security, diversity, and efficiency. We further introduce bank-health metrics, such as maximum exposure, maximum conditional exposure, adjusted effective bank size, and the rarely-administered fraction, all of which extend this picture of item utilization. AQuAP illustrates how operational analytics can translate psychometric concepts into quality assurance tools for high-volume, AI-enabled testing programs. This work is illustrated with the Duolingo English Test (DET) processes.