The paper starts by clarifying the terminology around out-of-distribution (OOD) detection in reinforcement learning, distinguishing between sensory anomalies (changes to observations) and semantic anomalies (changes to environment dynamics).
The authors then introduce three new benchmark environments - ARTS, ARNO, and ARNS - that contain temporally correlated anomalies, in contrast to previous benchmarks that focused on i.i.d. or time-independent anomalies. Experiments show that current state-of-the-art OOD detectors like PEDM struggle to identify these temporally correlated anomalies.
To address this, the authors propose a new detection method called DEXTER (Detection via Extraction of Time Series Representations). DEXTER first extracts a diverse set of time series features from the agent's observations, and then uses an ensemble of isolation forest models to compute anomaly scores. The authors also introduce DEXTER+C, which uses a CUSUM-based decision rule to classify episodes as OOD.
Evaluations show that DEXTER and DEXTER+C significantly outperform PEDM and other baselines on the new benchmark environments, both in terms of AUROC scores and the number of timesteps required to detect anomalies. The authors also find that DEXTER performs well on standard benchmark scenarios, though a combination of DEXTER and PEDM may yield optimal results.
The paper concludes by discussing the importance of addressing temporally correlated anomalies for the safe deployment of reinforcement learning agents in the real world, and outlines several directions for future work.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Linas Nasvyt... at arxiv.org 04-11-2024
https://arxiv.org/pdf/2404.07099.pdfDeeper Inquiries