Search papers, labs, and topics across Lattice.
This study introduces a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that addresses the asymmetric consequences of false negatives and false positives in ransomware detection by incorporating a tailored reward signal. By evaluating four deep reinforcement learning agents with both symmetric and asymmetric reward structures, the research demonstrates that the asymmetric approach significantly enhances detection performance, particularly in reducing missed detections. Key results indicate that the DDQN agent with the asymmetric reward achieved a false-negative rate of 0.0080 and an F1-score of 0.9915, marking a 67.6% reduction in missed detections compared to the best supervised baseline.
Asymmetric reward design in deep reinforcement learning can drastically reduce false negatives in ransomware detection, achieving a remarkable 67.6% improvement over traditional methods.
Ransomware detection is a security-critical task in which false negatives and false positives have unequal operational consequences. Conventional machine learning detectors often use symmetric objectives that penalize missed ransomware detections and benign false alarms equally, although a false negative can cause irreversible encryption, operational disruption, and high recovery cost, whereas a false positive is usually reversible. This study proposes a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that embeds false-negative and false-positive cost asymmetry into the reinforcement learning reward signal to prioritize missed-detection reduction. The framework also introduces a Security-Optimal Model Selection (SOMS) criterion and an adaptive episode-level sample-ordering mechanism. Four deep reinforcement learning agents, DQN, DDQN, PPO, and A2C, were evaluated using a symmetric baseline reward (R1) and a security-aware asymmetric reward (R2). Experiments used four discount factors, five-fold cross-validation, and three random seeds, resulting in 480 training runs on a balanced ransomware detection dataset. The SOMS criterion selects models by prioritizing false-negative rate, followed by F1-score and training time. Results show that asymmetric reward shaping improves security-oriented detection performance. The SOMS-selected configuration, DDQN with R2 and gamma = 0.1, achieved a false-negative rate of 0.0080, an F1-score of 0.9915, and an AUC of 0.998, reducing missed detections by 67.6% compared with the best supervised baseline. Across all configurations, R2 reduced the mean false-negative rate by 43% relative to R1. These findings show that reward-function design is important for security-sensitive ransomware detection.
