Automatic Construction of Contrastive Pairs for Improving Large Language Model Alignment

Automatically constructing contrastive pairs from outputs of large language models of varying capabilities (e.g., GPT-4, ChatGPT, InstructGPT) can effectively improve the alignment of large language models through contrastive post-training techniques like Direct Preference Optimization (DPO).