Is RLHF More Difficult than Standard RL? A Theoretical Perspective
Dec 10, 2023
Speakers
Yuanhao Wang
Speaker · 2 followers
Qinghua Liu
Speaker · 0 followers
About
Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifically, (1) for preferences that are drawn from reward-based probabilistic models, we reduce the problem to robust reward-based RL that can tolerate small errors in rewards; (2) for general arbitrary preferences where the objective is to find the von Neumann winner, we reduce the problem to multiagent reward-based RL which finds Nash equilibria for factored Markov games under a restricted set of policies. The latter case can be further reduce to adversarial MDP when preferences only depend on the final state. We instantiate all reward-based RL subroutines by concrete provable algorithms, and apply our theory to a large class of models including tabular MDPs and MDPs with generic function approximation. We further provide guarantees when K-wise comparisons are available.Reinforcement learning from Human Feedback (RLHF) learns from preference signals, while standard Reinforcement Learning (RL) directly learns from reward signals. Preferences arguably contain less information than rewards, which makes preference-based RL seemingly more difficult. This paper theoretically proves that, for a wide range of preference models, we can solve preference-based RL directly using existing algorithms and techniques for reward-based RL, with small or no extra costs. Specifica…
Organizer
NeurIPS 2023
Account · 617 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Unsupervised Image Denoising with Score Function
Watch later
Yutong xie, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Geometry-Informed Neural Operator
Watch later
Zongyi Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Watch later
Pei Chen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning Topology-Agnostic EEG Representations with Geometry-Aware Modeling
Watch later
Ke Yi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
An Efficient Dataset Condensation Plugin and Its Application to Continual Learning
Watch later
Enneng Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Unsupervised Protein-Ligand Binding Energy Prediction via Neural Euler's Rotation Equation
Watch later
Wengong Jin, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%