The paper introduces a novel method called Knowledgeable Agents from Language Model Rollouts (KALM) that leverages the knowledge embedded in large language models (LLMs) to enhance the skill acquisition of reinforcement learning (RL) agents. The key idea is to utilize the LLM to generate imaginary rollouts that capture a broader range of skills, including those not present in the original offline dataset, and then integrate these rollouts with offline RL to train more versatile and informed agents.
The main components of KALM are:
LLM Grounding: KALM fine-tunes the LLM to perform various tasks based on environmental data, including bidirectional translation between natural language descriptions of skills and their corresponding rollout data. This grounding process enhances the LLM's comprehension of environmental dynamics.
Rollout Generation: The fine-tuned LLM is then used to generate diverse and meaningful imaginary rollouts that reflect novel skills, including those that require unprecedented optimal behaviors.
Skill Acquisition: The offline RL training is conducted using both the original offline dataset and the LLM-generated imaginary rollouts, enabling the agent to acquire a broader set of skills.
The experiments on the CLEVR-Robot environment demonstrate the effectiveness of KALM. Compared to baseline offline RL methods, KALM achieves a significantly higher success rate (46% vs. 26%) on tasks with unseen natural language goals, showcasing its ability to generalize to novel situations. The results also highlight the LLM's capacity to comprehend environmental dynamics and generate meaningful imaginary rollouts that reflect novel skills, enabling the seamless integration of large language models and reinforcement learning.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Jing-Cheng P... at arxiv.org 04-16-2024
https://arxiv.org/pdf/2404.09248.pdfDeeper Inquiries