OmniACT: A Dataset and Benchmark for Multimodal Generalist Autonomous Agents

OmniACT introduces a dataset and benchmark for assessing agents' capability to generate executable programs for computer tasks, highlighting the challenge for conventional web agents.