Semantic Drift in Text Generation: Measuring and Mitigating the Decline in Factual Accuracy

Modern language models tend to generate correct facts initially, but then systematically drift away from the topic and generate incorrect facts later in the text. This "semantic drift" can be measured and mitigated through early stopping and reranking methods to improve the factual accuracy of generated text.