Memorization and Few-Shot Learning Capabilities of Large Language Models on Tabular Data

Large language models have memorized many popular tabular datasets verbatim, leading to inflated few-shot learning performance estimates on those datasets. However, they also exhibit non-trivial few-shot learning abilities on novel tabular datasets, which are largely driven by their world knowledge rather than in-context statistical learning.