The content discusses a novel approach to text generation that involves retrieving context-aware phrases from supporting documents. By addressing challenges in constructing training oracles, the method outperforms standard language models on various tasks and demonstrates improved quality in open-ended text generation.
Standard language models generate text using fixed vocabularies, while the proposed method selects context-aware phrases from supporting documents. Training oracles are initialized using linguistic heuristics and refined through self-reinforcement. Extensive experiments show superior performance over baselines, with increased accuracy and improved generation quality.
The approach transitions from token generation to phrase retrieval, enhancing interpretability and factuality of language models. A balanced design is emphasized, with source and target encoders for prefixes and phrases respectively. Efficient maximum inner product search algorithms are used for phrase retrieval.
Training objectives include InfoNCE loss for phrase retrieval and next-token prediction loss for token-level generation. Negative sampling techniques improve model differentiation ability, incorporating in-batch negatives and hard negatives to enhance discriminative representations.
Experiments demonstrate the effectiveness of the method on knowledge-intensive tasks like question answering and open-ended text generation. Results show significant improvements over standard LMs and state-of-the-art methods across various datasets.
Human evaluation scores indicate better performance of the proposed method in coherence, informativeness, fluency, and grammar compared to baseline models. The model also achieves high MAUVE scores with balanced coherence and diversity metrics.
Ablation studies highlight the importance of self-reinforcement mechanisms in enhancing text generation quality over multiple rounds of training iterations. The proposed approach offers scalability with domain-specific indices for improved performance without additional training.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Bowen Cao,De... at arxiv.org 03-01-2024
https://arxiv.org/pdf/2402.17532.pdfDeeper Inquiries