Search papers, labs, and topics across Lattice.
This study introduces MASH-Bench, a benchmark comprising 6,968 mass shooting incidents from four U.S. databases, to address the challenges of cross-source generalization in risk classification models. The evaluation reveals that while models like Random Forest and XGBoost achieve high recall on curated sources, their performance drastically declines on the Gun Violence Archive, highlighting the critical role of feature completeness in classification accuracy. Additionally, domain adaptation methods were tested, with DANN showing some improvement in recall, yet overall, the findings underscore that feature availability and label prevalence are more significant factors than the choice of classifier in cross-source settings.
Cross-source generalization in mass-shooting risk classification collapses due to feature deficiencies, revealing that model choice is less critical than data quality.
Public mass-shooting databases differ substantially in coverage, feature availability, and reporting practices, creating challenges for machine-learning models that must generalize across data sources. We introduce MASH-Bench, a harmonized benchmark of 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA). We evaluate cross-source risk classification using leave-one-dataset-out (LODO) evaluation. Random Forest, XGBoost, and LightGBM achieve VeryHigh-risk recall of 0.68-0.89 on the curated sources but generalize poorly to GVA, where mean recall drops to 0.20 and precision to 0.0004. To investigate the source of this degradation, we conduct a controlled feature-masking ablation that removes the five features unavailable in GVA from the curated sources. The resulting recall collapse to zero provides evidence that feature completeness is a major contributor to the observed cross-source failure. We further evaluate three domain-adaptation approaches: DANN, CORAL, and importance weighting. DANN improves VeryHigh-risk recall on GVA by 0.282 (95% CI [0.11, 0.47], p = 0.003), although precision remains low, whereas CORAL and importance weighting yield zero recall. Oracle prior-shift recalibration likewise fails to recover VeryHigh-risk predictions, indicating that label-side correction alone is insufficient under the observed feature deficiencies. A per-group audit further identifies substantial disparities associated with media-attributed mental-health labels. Overall, these results indicate that, in MASH-Bench, cross-source generalization is constrained more by feature completeness and label prevalence than by classifier choice. The benchmark provides a controlled setting for diagnosing these effects in cross-source risk classification.
