Khái niệm cốt lõi
Language models exhibit structural priming effects that can be explained by inverse frequency effects, such as prime surprisal and verb preference, as well as lexical dependence between prime and target.
Tóm tắt
The paper explores structural priming in language models, investigating which linguistic factors at the sentence and token level play an important role in influencing language model predictions. The authors make use of the structural priming paradigm, where recent exposure to a structure facilitates processing of the same structure, to examine whether and where priming effects occur in language models, and what factors predict them.
The key findings are:
Language models exhibit asymmetrical priming effects, where the strength and direction of the priming effect is often inverse to what is observed in humans. This asymmetry can be explained by inverse frequency effects, such as prime surprisal and verb preference.
Lexical overlap between prime and target, especially for verbs and function words, plays a strong role in balancing the priming effects and making them more consistent across structures.
Token-level analysis reveals that priming effects in language models are highly influenced by specific lexical items, and that they incorporate systematic properties of human production preferences learned from the training data.
Regression analysis shows that factors known to predict priming in humans, such as semantic similarity, surprisal, and verb preference, also predict priming effects in language models. This suggests that language models are able to pick up on abstract patterns influencing language predictions in humans.
Overall, the results provide insights into the mechanisms underlying structural prediction in language models and how they relate to human language processing.
Thống kê
The girl gave the ball to the boy.
The girl gave the boy the ball.
The baker gave the lady the cake.
Trích dẫn
"Structural priming is the phenomenon where speakers are more likely to repeat a certain structure after being recently exposed to a sentence containing a congruent structure."
"Priming effects in humans are typically stronger when there are shared words between prime and target, and when the prime is more unusual, or less frequent."
"Structural preference—which expresses within which structure a verb is most likely to occur—is another important factor when predicting priming behaviour."