Search papers, labs, and topics across Lattice.
This study introduces the Agentic BKT pipeline, a multi-agent LLM architecture designed to assess financial literacy stealthily during gameplay in serious games. By processing structured event logs from a 2D platformer game, the pipeline utilizes domain-specific agents to evaluate player decisions across financial competencies, ultimately synthesizing these assessments into a comprehensive mastery score. The results indicate that the multi-agent approach significantly enhances predictive validity compared to a single-LLM baseline, correlating mastery estimates with learning gains and post-test scores while demonstrating robust validity metrics.
A multi-agent LLM architecture can triple the predictive validity of financial literacy assessments in serious games, revealing the critical role of domain decomposition in capturing complex competencies.
Assessing financial literacy during gameplay without disrupting the learning experience remains a key challenge in serious games for education. We present the Agentic BKT pipeline, a multi-agent large language model architecture for stealth assessment of financial competencies from open-ended gameplay events. The pipeline processes events from a 2D platformer serious game aligned with the OECD/INFE financial literacy framework through four phases: (1) the game captures every player decision as a structured event log; (2) an LLM event classifier labels each action on a four-point rubric validated against three domain experts (Fleiss kappa = 0.624, substantial agreement); (3) four domain-specific agents specializing in risk mitigation, investing, spending, and credit management perform session-level reasoning over behavioral trajectories, feeding per-competency Bayesian Knowledge Tracing that estimates mastery within each domain; and (4) an expert judge agent synthesizes the domain-level estimates into an overall mastery score. Evaluated with 193 K-12 participants across 264 game sessions, the Agentic BKT pipeline yields mastery estimates significantly correlated with learning gain (r = 0.276, p = 0.0001) and post-test scores (r = 0.333, p < 0.0001) while showing no correlation with pre-test scores, providing both convergent and discriminant validity. The multi-agent approach approximately triples the predictive validity of a single-LLM baseline (r = 0.095, not significant) in this study, demonstrating that domain decomposition and session-level reasoning play a central role in capturing the multidimensional nature of financial literacy from gameplay
