Search papers, labs, and topics across Lattice.
This paper investigates instructional alignment in cybersecurity crisis preparation exercises by analyzing multimodal data from 23 teams, using Bloom's taxonomy to code objectives and team emails. Generalized linear mixed models reveal that the discrepancy between required and enacted Bloom levels (alignment) predicts team success, outperforming raw Bloom category scores. Furthermore, L1-regularized logistic regression demonstrates that text embeddings and log features derived from multimodal traces are superior predictors of performance compared to Bloom-only models, achieving a test AUC of 0.80.
Forget Bloom's taxonomy alone: the *mismatch* between intended and actual cognitive activity during cybersecurity exercises is what really predicts team success.
Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationalize at scale. We examine alignment in cybersecurity simulations using multimodal traces from 23 teams (76 students) across five exercise sessions. Study 1 codes objectives and team emails with Bloom's taxonomy and models the completion of key exercise tasks with generalized linear mixed models. Alignment, defined as the discrepancy between required and enacted Bloom levels, predicts success, whereas the Bloom category alone does not predict success once discrepancy is considered. Study 2 compares predictive feature families using grouped cross-validation and l1-regularized logistic regression. Text embeddings and log features outperform Bloom-only models (AUC~0.74 and 0.71 vs. 0.55), and their combination performs best (Test AUC~0.80), with Bloom frequencies adding little. Overall, the work offers a measure of alignment for simulations and shows that multimodal traces best forecast performance, while alignment provides interpretable diagnostic insight.
CMU ML