Adversarial Images Can Hijack and Control the Behavior of Vision-Language Models at Runtime

Adversarial images can be crafted to control the behavior of vision-language models at inference time, forcing them to generate arbitrary outputs, leak information, bypass safety constraints, and believe false statements.