Core Concepts
Proposing a few-shot adversarial prompt framework to enhance robustness in vision-language models by addressing key limitations.
Abstract
The content discusses the vulnerability of deep neural networks to imperceptible adversarial perturbations and introduces a few-shot adversarial prompt framework to improve robustness. It addresses issues with previous methods, such as heavy adaptation costs and suboptimal text supervision. The proposed framework leverages adversarially correlated text supervision and a novel training objective to enhance consistency of multi-modal features.
Directory:
- Abstract
- Discusses the vulnerability of deep neural networks to imperceptible adversarial perturbations.
- Introduces a few-shot adversarial prompt framework to improve robustness.
- Introduction
- Highlights the challenges posed by adversarial examples in misleading DNNs.
- Discusses the importance of semantic information for human cognition compared to statistical associations in machines.
- Method
- Introduces the Few-shot Adversarial Prompt learning (FAP) framework for adapting pre-trained VLMs in a few-shot manner.
- Describes learnable text supervision for adversarial examples and balancing natural and adversarial generalization.
- Experiments
- Evaluates the performance of the proposed method on various datasets, showcasing superior results in both natural and robust accuracy.
- Conclusion
- Summarizes the contributions of the research in enhancing model robustness against adversarial attacks.
Stats
"achieves zero-shot adversarial robustness by aligning adversarial visual features with text supervision."
"matches state-of-the-art zero-shot adversarial robustness with only 1% training data."
Quotes
"The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment."
"Our method matches the benchmark result with 1.25% examples from ImageNet, thus speeding up the training process."