Core Concepts
Optimizing user preferences through a framework combining language models and Bayesian Optimal Experimental Design.
Abstract
The content introduces OPEN, a framework that leverages language models and Bayesian Optimal Experimental Design to optimize user preferences. It addresses challenges in preference learning and outperforms existing methods in user studies. The framework involves featurization, initializing user preferences, selecting optimal questions, verbalizing queries, updating preferences, and prediction. Results show improved alignment with human preferences compared to LM-only approaches. The study evaluates OPEN in a content recommendation domain with 30 articles from the New York Times. User feedback highlights the importance of feature weightings and the challenges of self-mapping features to NL questions.
Stats
Aligning AI systems to users’ interests requires understanding and incorporating humans’ complex values and preferences.
Language models have been used to gather information about human preferences.
OPEN outperforms existing LM- and BOED-based methods for preference elicitation.
Quotes
"OPEN can optimize the informativity of queries while remaining adaptable to real-world domains."
"LMs are still subpar at in-context-learning of human preferences from demonstrations."