Search papers, labs, and topics across Lattice.
This paper introduces OOD-RL-Bench, a novel benchmark framework specifically designed for evaluating out-of-distribution (OOD) detection in reinforcement learning (RL) contexts, addressing the limitations of existing benchmarks that focus primarily on static datasets. By integrating detectors and anomaly injectors through shared interfaces, the framework allows for the assessment of various anomaly types within RL trajectories, revealing significant performance variances across different conditions. The evaluation, conducted using a Deep Q-Network in the LunarLander-v3 environment, highlights that while some anomalies like observation perturbations are detected with high accuracy, others such as action-conditioned dynamics pose substantial challenges for current detection methods.
Performance of OOD detectors varies dramatically across anomaly types, with some easily identified while others remain elusive, underscoring a critical gap in RL robustness.
Reliable reinforcement learning (RL) agents must maintain operational integrity amidst sensor malfunctions, dynamic disturbances, and slow environmental shifts. The detection of out-of-distribution conditions is pivotal to determining when an agent's observations, transitions, or trajectory dynamics deviate from the assumptions underpinning its policy training. Current out-of-distribution (OOD) detection benchmarks typically evaluate image classifiers or static low-dimensional datasets, failing to account for the complex, action-dependent temporal structure inherent in RL trajectories. To address this gap, we present OOD-RL-Bench, a comprehensive and extensible framework designed to evaluate OOD detectors against categories of anomalies injected into RL trajectories. Detectors and anomaly injectors are integrated through shared interfaces and configuration, which allows new scoring methods and perturbation families to be evaluated without modification of the core benchmark loop. We evaluate the utility of the framework using a Deep Q-Network policy within the LunarLander-v3 environment. We assess the performance of each detector across a suite of anomaly types using matched-time AUROC, matched-time AUPRC, matched-time false-positive rate, detection delay, and segmented-onset metrics. Our analysis reveals significant performance variance across anomaly types: observation perturbations and regime switches are identified with high accuracy by several methods, while observation delay and action-conditioned dynamics remain difficult even when post-onset anomaly scores are compared against clean scores from the same timesteps. We make the framework, trained policy checkpoint, and complete results publicly available as a reproducible artefact.
