核心概念
Pre-training transforms forgetful language models into retentive ones, influenced by knowledge relevance and diversification.
摘要
Memory is crucial for cognitive functions, with pre-trained language models showing remarkable memorizing abilities. Vanilla models suffer from catastrophic forgetting, while pre-training enhances memory retention. Knowledge relevance and diversification significantly impact memory formation.
統計資料
Memory is strengthened through repetitive learning.
Pre-training leads to retentive language models.
Knowledge relevance and diversification influence memory formation.
引述
"Vanilla language models are forgetful."
"Pre-training is at the core of the forgetful to retentive transformation."
"Knowledge relevance and diversification significantly influence memory formation."