Improving Predictive Performance in Probabilistic Programs with Stochastic Support by Optimizing Path Weights

Probabilistic programs with stochastic support can be decomposed into a weighted sum of local posteriors associated with each possible program path. This decomposition reveals that using the full posterior implicitly performs Bayesian model averaging (BMA) over the paths. However, BMA weights can be unstable due to model misspecification or inference approximations, leading to suboptimal predictions. To address this, the authors propose alternative mechanisms for path weighting based on stacking and PAC-Bayes objectives, which can be implemented as a cheap post-processing step on top of existing inference engines.