Controlling Smartphones with Vision-Language Models: Enabling Intuitive Device Interaction through Screen-Based Commands

A novel vision-language model capable of executing diverse user instructions on mobile devices by interacting solely with the user interface, leveraging both visual and textual inputs.