Dynamically Composing Large Language Model Policies to Solve Complex Web Tasks

Core Concepts

Dynamically composing a stack of large language model policies enables solving a diverse set of complex web tasks by adapting to the task complexity.

Abstract

The paper proposes Stacked LLM Policies for Web Actions (SteP), a framework that dynamically composes policies from a library to solve complex web tasks. The key insight is that a single LLM policy is insufficient to handle the combinatorially large open-world space of web tasks and variations across web interfaces. Instead, SteP defines a Markov Decision Process where the state is a stack of policies, allowing any policy to invoke any other policy dynamically.
The paper makes the following key contributions:

SteP framework that defines an MDP over a stack of policies, enabling dynamic composition to solve complex web tasks.
Experimental validation on WebArena, MiniWoB++, and an airline CRM simulator. SteP outperforms prior works on WebArena and is competitive on MiniWoB++ while using significantly less data.
SteP achieves higher success rates than baselines that use a single policy, by breaking down tasks into smaller subtasks handled by dedicated policies. This also results in lower overall context lengths and costs per trajectory.
In-context examples help associate language instructions with webpage elements, particularly for simplified pages with ambiguous UI. SteP leverages dedicated examples in each policy prompt more effectively than a single flat prompt.

Stats

The paper reports the following key metrics:

Success rate (suc↑) on WebArena tasks, ranging from 0.15 to 0.36 for SteP compared to 0.12 to 0.23 for baselines.
Number of actions (#act) taken on WebArena tasks, with SteP using 9.16 actions on average compared to 6.44 for the Flat-4k baseline.
Success rate (suc↑) on MiniWoB++ tasks, with SteP Few-shot achieving 0.96 compared to 0.72 for Flat Few-shot.

Quotes

"Performing tasks on the web presents fundamental challenges to large language models (LLMs), including combinatorially large open-world tasks and variations across web interfaces."
"Our key insight is to enable dynamic control, where any policy can choose to invoke any other policy. Such expressiveness is crucial for solving web tasks that require policies to operate at multiple levels of abstraction."
"SteP defines a Markov Decision Process (MDP) where the state is a stack of policies. The stack stores the dynamic control state capturing the chain of policy calls that evolves over time."

Key Insights Distilled From

SteP: Stacked LLM Policies for Web Actions

by Paloma Sodhi... at arxiv.org 04-24-2024

https://arxiv.org/pdf/2310.03720.pdf

Deeper Inquiries

How can the policy library be automatically expanded or updated based on user feedback or new web tasks?

Expanding or updating the policy library in the SteP framework can be automated through a few key techniques:

User Feedback Integration: Implement a feedback loop where users can provide input on the effectiveness of existing policies. This feedback can be used to identify gaps or areas for improvement in the current policies. For instance, users could flag tasks that were challenging to complete or suggest new tasks that are not covered by existing policies.

Active Learning: Utilize active learning techniques to identify new tasks or intents that the current policies are not equipped to handle. By actively seeking out new data points or scenarios where the model struggles, the system can automatically trigger the creation of new policies to address these gaps.

Transfer Learning: Implement transfer learning methods to adapt policies from similar tasks or domains to new tasks. By leveraging knowledge from existing policies and fine-tuning them on new data or tasks, the system can quickly expand the policy library without starting from scratch.

Clustering and Generalization: Use clustering algorithms to group similar tasks together and generalize policies to cover a broader range of tasks within each cluster. This approach can help in efficiently scaling the policy library by creating policies that can handle multiple related tasks.

Continuous Learning: Enable the system to continuously learn from new data and tasks encountered in real-world usage. By updating policies based on real-time interactions and performance metrics, the policy library can evolve and adapt to changing user needs and task requirements.

How could SteP be extended to handle partially observable environments or tasks that require long-term planning and reasoning about the global state, beyond the local context of individual policies?

To extend SteP to handle partially observable environments and tasks requiring long-term planning and reasoning about the global state, the following techniques could be employed:

Memory Augmented Policies: Integrate memory mechanisms into individual policies to store and retrieve relevant information over time. This allows policies to maintain a long-term memory of past interactions and observations, enabling them to make informed decisions based on historical context.

Hierarchical Planning: Implement a hierarchical planning framework where high-level policies can coordinate the actions of lower-level policies. This hierarchical structure allows for long-term planning by breaking down complex tasks into manageable subtasks and coordinating the execution of multiple policies towards a common goal.

Attention Mechanisms: Enhance policies with attention mechanisms that enable them to focus on relevant information across different time steps. By attending to critical details in the environment's history, policies can make informed decisions that consider the global state and long-term dependencies.

Reinforcement Learning with Temporal Abstraction: Incorporate reinforcement learning techniques that support temporal abstraction, allowing policies to operate at different levels of granularity. By learning to abstract actions over extended time horizons, policies can reason about the global state and plan actions that span multiple time steps.

State Aggregation: Develop mechanisms for aggregating information from multiple policies to construct a comprehensive global state representation. By combining local context from individual policies, the system can build a holistic view of the environment, facilitating long-term planning and decision-making.

By integrating these advanced techniques, SteP can be extended to handle complex tasks in partially observable environments, enabling it to reason about the global state and perform long-term planning effectively.