RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

08/08/2023

∙

To use reinforcement learning from human feedback (RLHF) in practical applications, it is crucial to learn reward models from diverse sources of human feedback and to consider human factors involved in providing feedback of different types. However, the systematic study of learning from diverse types of feedback is held back by limited standardized tooling available to researchers. To bridge this gap, we propose RLHF-Blender, a configurable, interactive interface for learning from human feedback. RLHF-Blender provides a modular experimentation framework and implementation that enables researchers to systematically investigate the properties and qualities of human feedback for reward learning. The system facilitates the exploration of various feedback types, including demonstrations, rankings, comparisons, and natural language instructions, as well as studies considering the impact of human factors on their effectiveness. We discuss a set of concrete research opportunities enabled by RLHF-Blender. More information is available at https://rlhfblender.info/.

READ FULL TEXT

RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback

Reward-rational (implicit) choice: A unifying formalism for reward learning

Which Spurious Correlations Impact Reasoning in NLI Models? A Visual Interactive Diagnosis through Data-Constrained Counterfactuals

Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework

Reinforcement Learning based Embodied Agents Modelling Human Users Through Interaction and Multi-Sensory Perception

RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

Related Research

Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback

Reward-rational (implicit) choice: A unifying formalism for reward learning

Which Spurious Correlations Impact Reasoning in NLI Models? A Visual Interactive Diagnosis through Data-Constrained Counterfactuals

Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

Facial Feedback for Reinforcement Learning: A Case Study and Offline Analysis Using the TAMER Framework

Reinforcement Learning based Embodied Agents Modelling Human Users Through Interaction and Multi-Sensory Perception