The paper proposes MagicPose, a novel approach for realistic human pose and facial expression retargeting. The key idea is to decompose the problem into two tasks: (1) identity/appearance control and (2) pose/motion control.
For appearance control, MagicPose introduces an Appearance Control Model that provides appearance guidance from a reference image to the Stable Diffusion (SD) model via a Multi-Source Attention Module. For pose control, MagicPose uses a Pose ControlNet to provide pose and expression guidance.
MagicPose employs a multi-stage training strategy to effectively learn these sub-modules and disentangle the appearance and pose control. Extensive experiments demonstrate MagicPose's ability to retain key features of the reference identities, including skin tone and clothing, while following the pose skeleton and facial landmark inputs. Moreover, MagicPose can generalize well to unseen identities and motions without any fine-tuning.
The paper makes the following key contributions:
To Another Language
from source content
arxiv.org
Principais Insights Extraídos De
by Di Chang,Yic... às arxiv.org 05-07-2024
https://arxiv.org/pdf/2311.12052.pdfPerguntas Mais Profundas