Temel Kavramlar
A novel system that can automatically extract semantically meaningful and replayable macros reflecting useful tasks on mobile apps from random or human-curated interaction traces.
Özet
The paper introduces a novel approach that uses Large Language Models (LLMs) with a trace-based chain-of-thought technique and optimal path synthesis to effectively extract large-scale, meaningful macros from interaction traces. The extracted macros are automatically tagged with natural language descriptions and are fully executable.
The key highlights of the approach are:
- Task Discovery: The LLM is used to discover potential user tasks on each UI screen in the interaction traces.
- Action Grounding: The LLM predicts the relevant actions for completing the discovered user tasks.
- Parameter-finding: The LLM predicts the additional information required for the user tasks.
- Action/Trace Merging: The system builds interaction graphs from multiple traces to find optimal paths for executing the macros.
The authors conduct multiple studies to validate the quality of the extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of the macros in live environments. The results demonstrate the effectiveness of the approach and the usefulness of the extracted macros in various downstream applications.
İstatistikler
The extracted macros from the RICO dataset contain 6.05 actions on average before optimization, and 3.41 actions after optimization, a 43.6% reduction.
The extracted macros from the Rehearsal dataset contain 7.51 actions on average before optimization, and 3.40 actions after optimization, a 54.7% reduction.
Alıntılar
"Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation."
"Recent advances in Large Language Models (LLMs) have enabled a new class of methods and models that can understand and interact with mobile apps with an unprecedented level of intelligence."