Leveraging Diffusion Models for Generalizable Dense Prediction

The core message of this paper is that by reformulating the diffusion process as a deterministic mapping between input images and output prediction distributions, and using low-rank adaptation to fine-tune pre-trained text-to-image diffusion models, the proposed DMP approach can effectively leverage the inherent generalizability of diffusion models to perform various dense prediction tasks, such as 3D property estimation, semantic segmentation, and intrinsic image decomposition, even with limited training data in a specific domain.