Pretraining a linearly parameterized single-layer linear attention model for linear regression with a Gaussian prior requires only a small number of independent tasks.
In-Context Learning (ICL) kann tatsächlich Beziehungen zwischen Etiketten aus den Beispielen in Kontexten lernen, ist aber nicht mit konventionellem Lernen gleichzusetzen.
High-capacity transformers can mimic Bayesian inference when performing in-context learning across a diverse range of linear and nonlinear function classes. The inductive bias of in-context learning is determined by the pretraining data distribution.
In-context learning (ICL) has emerged as a powerful paradigm for natural language processing, enabling large language models to make predictions based on a few demonstration examples. This survey aims to comprehensively review the progress and challenges of ICL.