Evaluating the Calibration of In-context Learning in Language Models

In-context learning (ICL) can lead to miscalibration, especially in low-shot settings, and methods aimed at improving usability like fine-tuning and chain-of-thought prompting can further degrade calibration.