核心概念
The author explores how combining split conformal prediction with Bayesian deep learning impacts out-of-distribution coverage, highlighting the importance of model confidence on calibration datasets.
摘要
The study investigates the effects of combining split conformal prediction with Bayesian deep learning on out-of-distribution coverage. It emphasizes the impact of model confidence levels on calibration datasets and provides practical recommendations for improving machine learning system safety. The research evaluates various inference techniques and offers insights into when conformal prediction may enhance or diminish out-of-distribution coverage.
統計資料
SGD is overconfident at both error tolerances, ENS is overconfident at 0.01 error tolerance, MFV, SGHMC, and LAPLACE are all underconfident.
Average set sizes vary across methods: SGD (1.20), ENS (1.34), MFV (2.97), SGHMC (1.87), LAPLACE (1.59) at 0.05 error tolerance.
For MedMNIST experiment: SGD credible set coverage - 94% at 0.05 error tolerance, 97% at 0.01 error tolerance.
引述
"We suggest that application of both methods in certain scenarios may be counterproductive and worsen performance on out-of-distribution examples."
"Understanding the interaction between predictive models and conformal prediction is crucial for safe deployment of machine learning systems."