Denoising Diffusion Models

Terminologies

Notations

Introduction

This article will introduce the definitions of DDPM and DDIM.

As stated earlier, the work of DDIM is based on DDPM.

DDPM

Diffusion Models often involves modeling two processes:

• forward process: noise data($x_{0}$) to data point($x_{t}$)
• reverse process: data point to noise data, the reversion of forward process

Forward Process

As a improvement of Diffusion Models, DDPM models the forward process as:

$$ \begin{equation} x_{t} = \alpha_{t} x_{t-1} + \beta_{t}\epsilon_{t}, \epsilon_{t} \sim \mathcal{N}(0,1), 0 \le t \le T, t \in \mathbb{Z} \end{equation} $$

where $\alpha, \beta > 0, \alpha_{t}^{2}+ \beta_{t}^{2} = 1$. This can be viewed as:

• the remains from the previous data: $\alpha_{t} x_{t-1}$
• the destruction by introducing noise $\epsilon_{t}$

Accordingly, the conditional probability of $x_t$ would be: $$ p(x_{t}| x_{t-1}) = \mathcal{N}(x_{t};\alpha_{t}x_{t-1}, \beta_{t}^{2}I) $$

[!NOTE]

• All $\alpha, \beta, T$ are constants
• Apparently, this is a Markovian process

Reverse Process

By applying the forward process for $T$ times, we have $t$ pairs of $(x_{t-1}, x_{t})$. This is our training data.

Reversing the forward process, the task of the reverse process should be: learn how to get $x_0$ from $x_{t}$, formally $x_0 \to x_{t}$.

DDPM splits this process into $t$ steps of $x_t \to x_{t-1}$.

[!TIP] The Methodology of DDPM DDPM is a Likelihood-based Model.

In the paper, they model each single step as a gaussian transition: $$ p_{\theta}(x_{t-1}|x_{t}) = \mathcal{N}(x_{t-1};\mu_{\theta}(x_{t}, t), \Sigma_{\theta}(x_{t},t)) $$ where:

• $\mu_{\theta}(x_{t}, t)$: the mean value
• $\Sigma_{\theta}(x_{t},t)$: the variance predictor (of reverse process).

[!NOTE]

• The noise is not gaussian noise multiplied by a factor, but predicted directly.
• Not to confuse: * $\Sigma_{\theta}(x_{t},t)$: the noise in reverse process. It has been tested positive to the reverse process * $\epsilon_\theta(x_{t},t) \to \epsilon_{t}$ : the noise predictor (of forward process)

Mean Value Predictor: $\mu_{\theta}(x_{t}, t)$

In order to model the $\mu_{\theta}(x_{t}, t)$, from Bayes’ Theorem we have: $$ p(x_{t-1}| x_{t},x_0)= \frac{p(x_t|x_{t-1}) p(x_{t-1} | x_0)}{p(x_{t}|x_{0}) } $$

The process of induction would be:

1. Predict $x_t$, $x_{t-1}$ from $x_0$
2. Replace all the variables($x_{0}$) with $x_{t}$ in Equation 4

Predict $x_{0}$ with $x_{t}$

Applying forward process $p(x_{t}|x_{t-1})$ for $t$ times, we can rewrite $x_{t}$ as:

$$ x_{t} = \bar \alpha_{t} x_{0} + \bar{\beta_{t}} ^ {2}\epsilon_{t}, \bar \alpha_{t} = \prod \alpha_{i}, \bar \beta_{t} = \sqrt{1-\bar \alpha_{t}^{2}} $$ And the probability version: $$ p(x_t|x_{0}) = \mathcal{N}(x_{t}; \bar \alpha_{t} x_{0}, \bar \beta_{t} ^ {2}I) $$

Now that we have $x_t$ from $x_0$, update Eq 4 (since $\mathcal{N}$ can be represented as probabilities, the result is conformed to $\mathcal{N}$ as well):

Let’s define the predicted mean value of $x_{t-1}$ as $\tilde \mu_t(x_{t}, x_{0}) = \frac{\alpha_{t}\bar \beta_{t-1}^{2}}{\bar \beta_{t}^{2}}x_{t} + \frac{\bar \alpha_{t-1}\beta_{t}^{2}}{\bar \beta_{t}^{2}}x_{0}$.

Notice the meaning of it: With Bayes’ Theorem, using $x_{t}$ and $x_{0}$, we can derive the mean value of $x_{t-1}$.

So naturally, we can make our $\mu_{\theta}(x_{t}, t)$, who have the same estimated output as $\tilde \mu_t(x_{t}, x_{0})$, learn the distribution of it: $$ \mu_{\theta}(x_{t}, t) = \tilde \mu_t(x_{t}, x_{0}) $$

[!NOTE] Different ways of modeling $\mu_\theta$ is also acceptable, it’s just that this is a better way (or not)

However, we don’t have $x_0$ to pass to $\tilde \mu_t(x_{t}, x_{0})$. Luckily, we can predict $x_0$ from rewriting Equation 7: $$ x_{0}= \frac{x_{t} - \sqrt{1- \bar \alpha_{t}}}{\sqrt{\bar \alpha_{t}}}\epsilon_{t} $$

[!TIP] This is actually an embodiment of the predictor–corrector method

Since we don’t have $\epsilon$ in the reverse process, we can make a neural work to learn it: $\epsilon_\theta(x_t,t) \to \epsilon_t$ : $$ x_{0}= \frac{x_{t} - \sqrt{1- \bar \alpha_{t}}}{\sqrt{\bar \alpha_{t}}}\epsilon_{\theta}(x_{t}, t) $$

Update the Eq 10: $$ \mu_{\theta}(x_{t}, t) = \tilde \mu_t(x_{t}, x_{0}) = \tilde \mu_t\left(x_{t}, \frac{x_{t} - \sqrt{1- \bar \alpha_{t}}}{\sqrt{\bar \alpha_{t}}}\epsilon_{\theta}(x_{t}, t)\right)= \frac{1}{\alpha_{t}}\left(x_{t} - \frac{\beta_{t}^2}{\bar\beta_{t}}\epsilon_{\theta}(x_{t},y, t)\right) $$

Reverse Noise Predictor: $\Sigma_{\theta}(x_{t},t)$

It still remains to design $\Sigma_{\theta}(x_{t},t)$, since it encourages diversity.

The DDPM paper suggested not learning it, since it:

resulted in unstable training and poorer sample quality

By fixing it at some value $\Sigma_{\theta}(x_{t},t) = \sigma_{t}^{2}I$ , where either $\sigma_{t}^{2} = \beta_{t}$ or $\tilde{\beta_t}$ yielded similar performance.

Training & Defining Loss

Conclusively, we have only defined one trainable model: $\epsilon_{\theta}(x_t, t)$

To reconstruct $x_{0}$

The training target can be MLE, the objective function being log-likelihood of reconstructing $x_{0}$:

To optimize the pixels

We design the loss function of $\theta$ as the Euclidean distance of the true and predicted mean of $x_{t-1}$: $$ \begin{align*} \ell &= ||x_{t-1} - \hat x_{t-1}|| ^ 2 \newline &= ||x_{t-1} - \mu_\theta(x_{t},t)|| ^ 2 \newline &=|| (\frac{1}{\alpha_{t}}(x_{t} - \beta_{t}\epsilon_{t}) ) ^ {2} - \frac{1}{\alpha_{t}}(x_{t} - \beta_{t}\epsilon_{\theta}(x_{t}, t)) ^ {2}||\newline &= \frac{\beta_{t}^{2}}{\alpha_{t}^{2}} ||\epsilon_{t} - \epsilon_{\theta}(x_{t}, t) || ^2 \end{align*} $$

DDIM

While the original DDPM is capable to generate satisfying images, it is known for poor performance: since the denoising(reverse) diffusion process usually take $T \sim 1000$ times of noise-prediction.

DDIM is proposed to boost the reverse process as Non-Markovian Process, by directly taking any model trained on DDPM and sampling only $T_{ddim}, T_{ddim} \le T$ timestamps, with some timestamps skipped. As a side-effect, the quality is compromised a little.

Reverse Process

It still takes the same approach as DDPM: predict $x_0$ from $x_t$ first.

From Eq 4, we can see that the sampling/training do involves $x_t$, but doesn’t actually involves $p(x_{t-1}|x_{t})$ (which is defined in our reverse model). Instead, it defines:

$$ q_\sigma(x_{t-1}|x_{t}, x_0) = \mathcal{N}(x_{t-1};\sqrt{\bar \alpha_{t-1}}x_0 + \sqrt{1 - \bar \alpha_{t-1} - \sigma_t^2}\frac{x_t - \sqrt{\bar \alpha_t}x_0}{\sqrt{1 - \bar \alpha_t}}, \sigma_t^2I) $$

Hence, the relation between $x_{t-1}$ and $x_t$ is:

$$ x_{t-1} = \sqrt{\alpha_{t-1}} \underbrace{\left( \frac{x_t - \sqrt{1 - \alpha_{t}}\epsilon_{\theta}(x_{t},t)} {\sqrt{\bar \alpha_{t}}} \right)}_{\text{predicted $x_0$}} + \underbrace{\sqrt{1 - \bar \alpha_{t-1} - \sigma_{t}^{2}\epsilon_{t}(x_{t},t)}}_{\text{predicted noise}} + \underbrace{\sigma_{t} \epsilon_{t}}_{\text{random noise}} $$ where: $$ \sigma_t = \eta \sqrt{(\frac{1 - \bar \alpha_t}{1 - \bar \alpha_{t-1}}) \left(1 - \frac{\bar \alpha_t}{\bar \alpha_{t-1}}\right)} $$ where $\eta \in (0,1)$ is a constant, indicating the level of random noise:

• $\eta = 1$: The random noise is maximized, which is DDPM.
• $\eta = 0$: The random noise is totally removed, making it a deterministic process/Implicit model, which is DDIM. It relies entirely on the predicted noise, while sacrificing some diversity with lowering random noise level.

As for the timestamps chosen, they are determined empirically.

<p class="notice-title">
    <span class="icon-notice baseline">
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="300.5 134 300 300">

    </span>Tip</p><p>In fact, $\eta$ represents the degree of moving some of the noise from predicted noise $\epsilon_t$ to sampled noise $\epsilon$: the bigger the $\eta$, the less deterministic, the larger random noise will be introduced to the reverse process.</p></div>