Efficient In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Tuning

The authors propose FT-PFN, a novel surrogate model for freeze-thaw Bayesian optimization that leverages in-context learning to efficiently and reliably extrapolate learning curves, outperforming existing deep Gaussian process and deep ensemble surrogates. When combined with their novel acquisition mechanism (MFPI-random), the resulting in-context freeze-thaw BO method (ifBO) yields new state-of-the-art performance on deep learning HPO benchmarks.