Improving Diffusion Models Through Learned Adaptive Noise Injection

Core Concepts

Learning the noise injection process in diffusion models, specifically using a multivariate and input-adaptive approach called MULAN, leads to improved log-likelihood estimation, faster training, and state-of-the-art density estimation performance on image datasets.

Abstract

Sahoo, S. S., Gokaslan, A., De Sa, C., & Kuleshov, V. (2024). Diffusion Models With Learned Adaptive Noise. Advances in Neural Information Processing Systems, 38.

This research paper investigates whether the noise injection process in diffusion models can be learned from data to improve log-likelihood estimation and probabilistic modeling. The authors aim to challenge the assumption that the ELBO (Evidence Lower Bound) is invariant to the choice of the diffusion process.

Key Insights Distilled From

Diffusion Models With Learned Adaptive Noise

by Subham Sekha... at arxiv.org 11-12-2024

https://arxiv.org/pdf/2312.13236.pdf

Deeper Inquiries

How might the principles of MULAN be applied to other generative modeling techniques beyond diffusion models, such as GANs or VAEs?

While MULAN is specifically designed for diffusion models, its core principles could potentially be adapted to other generative modeling techniques like GANs and VAEs. Here's how:
GANs:

Adaptive Noise Injection in the Generator: Instead of feeding fixed Gaussian noise to the GAN generator, we could incorporate a learned, data-dependent noise injection mechanism inspired by MULAN. This mechanism could learn to inject noise differently across different spatial regions or feature maps of the generated image, potentially leading to more realistic and diverse samples.
Context-Aware Discriminator: Similar to how MULAN uses auxiliary latent variables to condition the noise schedule, we could train a GAN discriminator that also takes these latent variables as input. This would allow the discriminator to learn more fine-grained distinctions between real and generated data based on high-level semantic information.
VAEs:

Learned Variance in the Encoder: VAEs typically assume a simple Gaussian prior over the latent space. We could instead learn a more complex, data-dependent variance structure in the encoder, similar to MULAN's multivariate noise schedule. This could allow the VAE to better capture the underlying data distribution and generate higher-quality samples.
Auxiliary Variable Conditioning: Similar to MULAN, we could introduce auxiliary latent variables to condition both the encoder and decoder of a VAE. This would enable the VAE to learn more disentangled and semantically meaningful latent representations, potentially improving sample quality and controllability.
Challenges:
Adapting MULAN's principles to GANs and VAEs presents challenges:

Training Instability: GANs are known for their training instability, and introducing learned noise schedules could exacerbate this issue. Careful regularization and architecture design would be crucial.
Latent Space Interpretation: In VAEs, interpreting the learned variance structure or auxiliary latent variables might be difficult. Techniques for visualizing and understanding these learned components would be essential.

Could focusing solely on log-likelihood optimization in diffusion models lead to unintended consequences or limitations in terms of the diversity and creativity of generated samples?

Yes, solely focusing on log-likelihood optimization in diffusion models could lead to unintended consequences:

Overfitting to the Data Distribution: Optimizing solely for log-likelihood might encourage the model to overfit to the training data, capturing even minor statistical noise. This could limit the diversity and creativity of generated samples, as the model might struggle to extrapolate beyond the seen data.
Mode Collapse: In pursuit of high likelihood, the model might focus on a limited subset of high-probability modes in the data distribution, neglecting other potentially interesting but less frequent modes. This phenomenon, known as mode collapse, can result in less diverse and less representative generated samples.
Lack of Exploration: An excessive focus on likelihood might discourage the model from exploring novel and creative regions of the data space that have low probability under the training distribution but could lead to interesting and unexpected samples.
Mitigations:
Several techniques can mitigate these potential downsides:

Regularization: Techniques like dropout, weight decay, and data augmentation can help prevent overfitting and encourage the model to learn more generalizable representations.
Diversity-Promoting Objectives: Incorporating additional loss terms that explicitly encourage diversity in the generated samples, such as perceptual losses or adversarial losses, can help counteract mode collapse and promote exploration.
Sampling Strategies: Exploring different sampling strategies, such as introducing noise during sampling or using techniques like top-k sampling, can help generate more diverse and creative samples.

If we view the noise injection process as a form of "exploration" in the data space, what insights can MULAN's learned noise schedules offer about the underlying structure of the data being modeled?

Viewing noise injection as "exploration," MULAN's learned noise schedules provide intriguing insights into the data structure:

Feature Sensitivity: The fact that MULAN learns to inject noise differently across different pixels or features suggests that the model discovers varying levels of importance or sensitivity among these features. Regions receiving less noise might represent more semantically salient or structurally defining aspects of the data, while those receiving more noise might be less critical for preserving identity or overall structure.
Hierarchical Information Flow: The use of auxiliary latent variables to condition the noise schedule hints at a hierarchical information flow during data generation. The latent variables might capture high-level semantic concepts, while the noise schedule modulates how these concepts translate into finer-grained details, reflecting the hierarchical organization of information in the data.
Data Manifold Geometry: The learned noise schedules could provide insights into the geometry of the underlying data manifold. Regions where the noise schedule changes rapidly might correspond to areas of high curvature or complex transitions in the manifold, while smoother schedules might indicate flatter regions.
Further Exploration:

Visualizing Noise Schedules: Visualizing the learned noise schedules directly, either as heatmaps over images or as trajectories in feature space, could reveal interpretable patterns and offer a more intuitive understanding of how the model explores the data space.
Analyzing Schedule Dynamics: Analyzing how the noise schedules evolve during training could shed light on how the model progressively learns the data structure. Early on, the schedules might be more uniform, becoming more specialized as the model discovers salient features and relationships.
Relating Schedules to Data Properties: Investigating how the learned schedules correlate with specific data properties, such as object categories, textures, or viewpoints, could uncover how the model adapts its exploration strategy based on semantic content.