Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for Reinforcement Learning from Human Feedback (RLHF) under Kullback-Leibler (KL) Constraint
This paper presents a theoretical framework for the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF), formulated as a reverse-KL regularized contextual bandit problem. It provides comprehensive theoretical analysis in offline, online, and hybrid settings, and proposes novel algorithms that incorporate uncertainty estimation and non-symmetric exploration structures to handle the KL penalty and preference learning challenges. The proposed methods significantly outperform existing baselines in real-world large language model experiments, showcasing the connections between solid theoretical foundations and powerful practical implementations.
MA-RLHF: Improving Reinforcement Learning from Human Feedback in Large Language Models Using Macro Actions
MA-RLHF improves the efficiency and quality of aligning large language models with human preferences by incorporating macro actions, which are sequences of tokens, into the reinforcement learning process.
REFUEL: An Efficient Policy Optimization Algorithm for Multi-Turn RLHF in Large Language Models
REFUEL is a novel and efficient algorithm for training large language models on multi-turn tasks using RLHF, addressing the covariate shift problem inherent in single-turn methods by employing on-policy data and a regression-based approach to predict relative future rewards.
Reinforcement Learning with Imperfect Corrective Actions and Proxy Rewards: An Iterative Approach
Combining imperfect proxy rewards with potentially suboptimal human corrective actions in a reinforcement learning framework can lead to more efficient learning and better-aligned policies compared to using either signal alone.
RLHFにおける精度のパラドックス：より良い報酬モデルが、より良い言語モデルを生み出さない場合がある
報酬モデルの精度が高ければ高いほど、常に言語モデルのパフォーマンスが向上するとは限らない。
On the Global Convergence of Online Reinforcement Learning from Human Feedback with Neural Network Parameterization
This research paper presents the first globally convergent online RLHF algorithm with neural network parameterization, addressing the distribution shift issue and providing theoretical convergence guarantees with state-of-the-art sample complexity.
