OmniACT: A Dataset and Benchmark for Multimodal Generalist Autonomous Agents

Virtual agents can automate computer tasks, but current models struggle with visual understanding.