The article presents a framework called PRELUDE (PREference Learning from User's Direct Edits) for interactive learning of language agents based on user edits to the agent's output. In a typical setting such as writing assistants, users interact with a language agent to generate a response, and may edit the agent's response to personalize it based on their latent preference.
The key insights are:
User edits provide natural feedback that can be leveraged to improve the agent's alignment with the user's preference, without the need for expensive explicit preference collection.
User preference can be complex, subtle, and context-dependent, making it challenging to learn. The article proposes a simple yet effective algorithm called CIPHER (Consolidates Induced Preferences based on Historical Edits with Retrieval) to address this.
CIPHER infers a textual description of the user's latent preference for a given context by leveraging a large language model (LLM) and retrieving similar past contexts. This learned preference is then used to generate future responses, leading to lower user edit costs over time.
Compared to baselines that directly use past user edits or learn context-agnostic preferences, CIPHER achieves the lowest cumulative user edit cost on two interactive writing assistant tasks - summarization and email writing. It also has lower computational expense than baselines that do not learn preferences.
The authors' analysis shows that the preferences learned by CIPHER have significant similarity to the ground truth latent preferences, demonstrating its effectiveness in capturing user needs.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Ge Gao,Alexe... at arxiv.org 04-24-2024
https://arxiv.org/pdf/2404.15269.pdfDeeper Inquiries