Optimal Ridge Regularization for Out-of-Distribution Prediction: Characterizing the Behavior of Regularization and Risk
The optimal ridge regularization level and the corresponding optimal risk can exhibit surprising behavior, such as negative regularization and non-monotonic risk profiles, especially in the out-of-distribution setting where the test distribution deviates from the training distribution.