The Impact of Near Duplicate Subwords on Language Model Efficiency
Near duplicate subwords in language model vocabularies can negatively impact training efficiency, but merging them may not yield the expected performance improvements.
Latent Distance Guided Alignment Training: Enhancing Large Language Models without Relying on Human Annotations
A novel DPO-based approach, LD-Align, that aligns a fine-tuned large language model with a high-quality supervised fine-tuning dataset without requiring any additional human annotations or relying on a more powerful language model.
Tailoring Synthetic Data for Aligning Large Language Models with Diverse Instruction Distributions
CodecLM, a framework that leverages large language models as codecs to generate high-quality synthetic data tailored for aligning target language models with diverse instruction distributions.
Preference Poisoning: Manipulating Language Models through Injected Poisoned Preference Data
An attacker can manipulate the behavior of a language model trained with RLHF by injecting a small amount of poisoned preference data into the training process, causing the model to generate more text containing a target entity in a desired sentiment.
Recursive Training on Synthetic Data Leads to Language Model Collapse: A Statistical Analysis
Recursive training on synthetic data generated from previous language models inevitably leads to model collapse, where the trained models lose diversity and converge to Dirac distributions. Incorporating a sufficient amount of real data can help mitigate this issue.
Robust Preference Optimization with Provable Noise Tolerance for Training Large Language Models
The key idea of ROPO is to dynamically assign conservative gradient weights to response pairs with high label uncertainty, based on the log-likelihood margins between the responses. This weighting strategy effectively suppresses the gradients of noisy samples and ensures that the expected risk maintains the same gradient direction under both noisy and noise-free conditions.
Aligning Large Language Models to Quote Verbatim from High-Quality Pre-Training Data for Improved Verifiability
Developing a method called QUOTE-TUNING that aligns large language models to quote verbatim from high-quality pre-training data, enabling more verifiable and truthful generations.
Direct Nash Optimization: A Scalable Algorithm for Aligning Large Language Models with General Preferences
Direct Nash Optimization (DNO) is a provable and scalable algorithm that optimizes large language models to align with general preferences, outperforming reward-based approaches and achieving state-of-the-art results.
CONSCENDI: A Contrastive and Scenario-Guided Approach to Generating Diverse Training Data for Guardrail Models in Virtual Assistants
CONSCENDI is a data generation pipeline that leverages scenario-guided conversations and contrastive examples to train smaller language models as effective guardrail models for virtual assistants. These guardrail models can identify rule violations in conversations with high accuracy, outperforming larger language models like GPT-4.
Training Large Language Models on Neurally Compressed Text: Challenges and Opportunities
Training large language models (LLMs) directly over highly compressed neural text can confer advantages in training and serving efficiency, as well as easier handling of long text spans. However, strong compression tends to produce opaque outputs that are not well-suited for learning by standard LLMs. The authors propose a novel compression technique called Equal-Info Windows that enables effective learning over neurally compressed text, outperforming byte-level baselines on perplexity and inference speed benchmarks.
© 2024 by Linnk AI