The paper presents a novel Semantic Image Synthesis (SIS) method that incorporates identity information during the generation process. The key idea is to extract an identity embedding from a pre-trained face recognition model and use it as an additional style code, which is then injected into the generator through a cross-attention mechanism.
The authors show that this identity injection has two main effects:
Identity Preservation: When using the same identity as the input face, the proposed method significantly improves the preservation of the original identity in the generated face, outperforming state-of-the-art SIS approaches.
Adversarial Attacks: When swapping the identity embedding with that of a different individual, the model can perform inconspicuous adversarial attacks on face recognition systems. The generated face will visually resemble the original subject, but will be recognized by the FR system as the target identity.
The authors conduct extensive experiments to validate these two capabilities. For identity preservation, they report improved cosine similarity scores between the original and generated faces across multiple face recognition models. For the adversarial attacks, they achieve state-of-the-art Attack Success Rates, while maintaining low perceptual differences between the original and attacked faces.
The paper also explores the effect of swapping different facial attributes (e.g., eyes, eyebrows, mouth) on the adversarial attack performance, finding that targeting identity-related regions leads to the most effective and inconspicuous attacks.
Overall, the proposed architecture demonstrates the ability to both preserve identity during semantic face generation and leverage this capability for powerful yet stealthy adversarial attacks on face recognition systems.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Giuseppe Tar... at arxiv.org 04-17-2024
https://arxiv.org/pdf/2404.10408.pdfDeeper Inquiries