Search papers, labs, and topics across Lattice.
This study evaluates the adversarial robustness of five smishing detection models, comparing three classical lexical models with two multilingual transformers against various attack types. The findings reveal that classical models suffer catastrophic failures under adversarial conditions, with a Robustness Degradation Ratio (RDR) reaching 0.988, while transformers show greater resilience, particularly mBERT, which has a maximum RDR of 0.351. This research highlights that performance on clean text does not reliably predict a model's robustness against adversarial attacks, emphasizing the need for architecture-specific defenses in cybersecurity applications.
Classical smishing detection models can fail catastrophically under adversarial attacks, while transformer-based models exhibit surprising resilience, challenging assumptions about clean-text performance.
Smishing detection systems are commonly trained and evaluated on clean, monolingual text. In low-resource settings, however, attackers frequently circumvent these systems through character obfuscation, cross-lingual code-switching, and structural perturbation. This study evaluates adversarial robustness for five model architectures: three classical lexical models (Random Forest, XGBoost, CNN+BiLSTM) and two multilingual transformers (mBERT, XLM-RoBERTa), using a dataset of 27,037 messages. Classical models are subjected to black-box generic attacks, while transformers are evaluated with attention-guided targeting. Each model is tested across three attack types and intensity levels, with performance measured by the Robustness Degradation Ratio (RDR). The results reveal a distinct architectural boundary: classical models experience near-catastrophic failure under character obfuscation and structural perturbation (RDR up to 0.988), whereas transformers demonstrate significantly greater resilience (RDR up to 0.351), with structural perturbation representing their most pronounced vulnerability. Effect-size analysis (Cliff's d) indicates a substantial difference between the two model categories. Within the transformer group, XLM-RoBERTa, despite achieving a higher clean-text baseline, exhibits greater degradation than mBERT. These findings demonstrate that clean-text performance is not a reliable predictor of adversarial robustness. Statistical validation using Mann-Whitney U and Friedman tests confirms that these patterns are attributable to model architecture rather than sampling. The results underscore the necessity for architecture-specific defences and frame smishing detection as an adversarial cybersecurity challenge rather than a static classification task.
