# CS 462 - Lecture 22

## <div class="clone-word"> **Generative Models** </div>

Bernhard Firner

2026-04-16

---

## Review

* We started talking about generative models
  * But didn't quite make it through GANs
* So we'll continue today, and hopefully talk about diffusion models as well

---

## Review: Evaluation

* How can we tell if fake data is good?
* **coverage**: Is my generator creating the same variety of data as the real data?
* **quality**: Does my generated data look like the real data?
* **disentangled latent space**: Are components of data independent?
  * For example, is the pose of my fake cat independent of the cat's fur pattern?
* **smooth response to z**: Does the output change smoothly from one reasonable image to another  as z is changed?

---

## Inception and FID

* Some of those evaluation metrics are difficult to measure
  * So instead we can try to make a DNN do the work
* We could simply classify our generated data (inception distance)
* Or we could compare the distance between features of real and fake data (Frechét Inception Distance)

---

## GANs

* GANs do a good job satisfying our quality metrics
  * Once we've employed a few techniques
* There are two components to GAN training
  * A *generator* creates fake images, given a latent vector, $z$
  * The *discriminator* classifies images as real or fake.
* The first major problem with GANs is to make a loss function that is stable

---

## Solution

* The generator tries to make the discriminator less certain that real images are real and more certain that fake images are real
  * If the two reach equilibrium, that the discriminator would misclassify both real and fake images
* Let's introduce a critic network, called C, that outputs the "realness" of an image

---

## Using a Critic

* Let's say $D(x) = sigmoid(C(x))$
  * The relativistic loss is based upon $D(\tilde{x}) = sigmoid(C(x_{real}) - C(x_{fake}))$
* The generator uses a similar loss, maximizing: $sigmoid(C(x_{fake}) - C(x_{real}))$

---

## In Batches

* In batches we compare each sample to the average critic output
  * For the discriminator, we want the real scores to be better than the fake scores:
  * $sigmoid(C(x_{real}) - \mathbb{E}[C(x_{fake})])$
* For the generator, we want the opposite
  * $sigmoid(C(x_{fake}) - \mathbb{E}[C(x_{real})])$
* And that is The Relativistic average GAN (RaGAN) as described in [a paper](https://arxiv.org/pdf/1807.00734)

---

## Modern GAN Training

* The GAN recipe was updated again in 2024
* [The GAN is dead; long live the GAN! A Modern GAN Baseline](https://proceedings.neurips.cc/paper_files/paper/2024/hash/4e2acb1e1c8e297d394ae29ed9535172-Abstract-Conference.html)
  * [https://github.com/brownvc/R3GAN](https://github.com/brownvc/R3GAN)
* The previous reformulation improves the loss surface, but it does not guarantee that a minima is actually found
* Two gradient penalties can improve training consistency
  * Basically, these try to prevent "overlearning" that would prevent oscillation

---

### Not People Examples

<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/r3gan-not-people.png" />
<br/>
<small>FFHQ2 data. See <a href="https://github.com/brownvc/R3GAN">the github</a> for instructions.</small>
</div>

---

## Not Birds Examples

<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/r3gan-not-birds.png" />
<br/>
<small>ImgNet data. See <a href="https://github.com/brownvc/R3GAN">the github</a> for instructions.</small>
</div>

---

## Conditional Generation

* There are other ways to train a GAN, of course
* For example, what if we want to control the output?
* We can introduce a second vector, $c$, to condition the output

</div>
<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/UDL/Chap15/GanConditional.svg" />
<br/>
<small>See the UDL text, Figure 15.13.</small>
</div>
</div>

---

## InfoGAN

* InfoGAN has the discriminator estimate c
  * We want c to end up with any latent information about the image
* Why? The latent variable, z, should be left with random noise and will be impossible to guess
  * So whatever is predicted in c must correspond to structured attributes
* By mixing continuous and discrete variables in $c$, they can be used to predict different facets of the data
* And you can [train your own](https://github.com/Natsu6767/InfoGAN-PyTorch)

-v-

## Aside

* Learning i.i.d. noise and learning structure have different bounds
* [Shannon's Source Coding Theorem](https://en.wikipedia.org/wiki/Shannon's_source_coding_theorem) tells us that, when we compress i.i.d. data, the minimal compression of information scales with the entropy
* Conversely, if what we are compressing is not i.i.d, [Kolmogorov complexity](https://en.wikipedia.org/wiki/Kolmogorov_complexity) is a better estimate
  * The Wikipedia summary is that the Kolmogorov complexity is the length of a shortest computer program that produces the object
  * Just insert "DNN" for "computer program" and that describes what we are doing

---

## InfoGAN Examples

<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/UDL/Chap15/GanInfoGAN.svg" />
<br/>
<small>See the UDL text, Figure 15.15.</small>
</div>

---

## StyleGAN

* [StyleGAN](https://openaccess.thecvf.com/content_CVPR_2019/html/Karras_A_Style-Based_Generator_Architecture_for_Generative_Adversarial_Networks_CVPR_2019_paper.html) separates high-level attributes of the latent space in an unsupervised manner
  * This improves disentangling, which was one of the metrics that we cared about
* This is accomplished by training a new set of control vectors, w
* How?
  * The initial layer begins with a constant representation, rather than noise and $z$
  * The $z$ is then mapped to multiple styles, via intermediate vectors, $w$

---

## Learning Styles

* $z$ is mapped to a new, multisegment vector w:
  * $f(z) \rightarrow w$
* After each convolution, noise is added to the partial synthetic image along with a style vector, taken from $w$
* So how is meaning imparted to $w$ through the loss?

</div>
<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/UDL/Chap15/GanStyleGANArch.svg" />
<br/>
<small>See the UDL text, Figure 15.19.</small>
</div>
</div>

---

## Learning Styles

* $w$ is converted into multiple style vectors, $y = (y_s, y_b)$
* Those influence the current model representation through an adaptive instance normalization layer
  * $AdaIN(x_i, y) = y_{s,i}\frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$
* Thus the style vectors control the strength of various features

---

## Mixing Regularization

* To teach the generator that styles are separate, sometimes two latent $z$ values are created, leading to two style vectors, $w_1$ and $w_2$
* Some styles are taken from $w_1$ and others from $w_2$ during generation, so G must separate styles at different depths
* This doesn't tell you *what* the style vectors will influence
  * But styles closer to the end of the network can only change small scale features
  * And styles closer to the beginning influence large features of the image

---

## Style GAN

* The components of $w$ can be thought of as coarse, medium, and fine, depending upon where they influence the output
* This leads to great control over the output images, as can be seem from example images in the [paper](https://openaccess.thecvf.com/content_CVPR_2019/html/Karras_A_Style-Based_Generator_Architecture_for_Generative_Adversarial_Networks_CVPR_2019_paper.html)

</div>
<div class="col">
<img style="width: 100%" class="r-stretch" src="./figures/UDL/Chap15/GanStyleGANArch.svg" />
<br/>
<small>See the UDL text, Figure 15.19.</small>
</div>
</div>

---

## Noise Vs Control

* If the style control are all non-stochastic, that means z is left with only stochastic effects
  * But what does that mean?
* The authors showed that stochasticity controlled:
  * the structure of arbitrary background objects
  * random features in the hair, leading to a "painterly" image if $z = 0$
  * realism of skin textures

---

## More Modern StyleGAN

* But what if we want more control?
* StyleGAN is from late 2018/early 2019, but it has been updated multiple times
* From 2022: [Third Time's the Charm? Image and Video Editing with StyleGAN3](https://arxiv.org/abs/2201.13433)
  * [Demo/github page](https://yuval-alaluf.github.io/stylegan3-editing/)
  * [Code github](https://github.com/yuval-alaluf/stylegan3-editing)

---

## Advances: Alignment

* Aligned Vs Unaligned Images
  * Some operations are easier learned when all images are aligned
  * So there are models to convert alignment and then apply the GAN
  * Then, rotation and translation are encoded into $z$ so that the original alignment can be restored

---

## Advances: Editing Directions

* StyleGAN uses a vector, S, to insert a style into the synthetic image
* [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://openaccess.thecvf.com/content/ICCV2021/html/Patashnik_StyleCLIP_Text-Driven_Manipulation_of_StyleGAN_Imagery_ICCV_2021_paper.html), from 2021, showed that S could be manipulated using text prompts and CLIP
  * The goal is to get a new latent code, $w$, given a source code, $w_s$
* Here comes some more math!

---

## Advances: Editing Directions

* So optimize this:
  * $\underset{w \in \mathcal{W}+}{argmin}D_{CLIP}(G(w), t) + \lambda_{L2} ||w - w_s|| + \lambda_{ID} \mathcal{L}_{ID}(w)$
  * $D_{CLIP}(G(w), t)$ is the cosine distance between the embedding of the Generator's output and the text embedding
  * ${L}_{ID}$ is a measure of output image similarity
* Basically, find a new $w$ that changes the images a little as possible but brings the CLIP embedding as close as possible

---

## Examples

* Again, the paper, github, and demo website have plenty of examples
* From 2022: [Third Time's the Charm? Image and Video Editing with StyleGAN3](https://arxiv.org/abs/2201.13433)
  * [Demo/github page](https://yuval-alaluf.github.io/stylegan3-editing/)
  * [Code github](https://github.com/yuval-alaluf/stylegan3-editing)
* But how is this useful?

---

## Applications

* Meme-worthy
  * [GAN-Supervised Dense Visual Alignment](https://openaccess.thecvf.com/content/CVPR2022/html/Peebles_GAN-Supervised_Dense_Visual_Alignment_CVPR_2022_paper.html) from 2022
  * Demos [here](https://github.com/wpeebles/gangealing)
* And more serious
  * [Expert-Guided StyleGAN2 Image Generation Elevates AI Diagnostic Accuracy for Maxillary Sinus Lesions](https://dl.acm.org/doi/abs/10.1145/3651781.3651810)
  * [Assessing the Efficacy of StyleGAN 3 in Generating Realistic Medical Images with Limited Data Availability](https://dl.acm.org/doi/abs/10.1145/3651781.3651810)

---

## Current Events

* GANs are still useful to generate new data in areas where data is limited
  * Medical fields, for example
  * A human expert can judge the image quality, but would have difficulty making images on their own
* There are newer versions of pix2pix and CycleGAN [on github](https://github.com/GaParmar/img2img-turbo)
* If you look, you'll see that they mention a bunch of new techniques
  * Like diffusion models!

---

## GAN Latent Spaces

* $z$, the latent vector, began as meaningless noise
* InfoGAN (and other conditional GANs) added more information
  * The discriminator had to be able to deduce a "realistic" context vector
  * That imposed meaning into the formerly random latent input
* With StyleGAN, the location that the context is inserted constrains what it can effect
* Finally, we can use an external model to provide additional constraints or to give direction to the style vectors

---

## Other Approaches

* GANs are fast and give good results
  * But training can be difficult sometimes
  * Mode collapse also means that the generator may avoid "difficult" areas of the data distribution
  * Control has tended to be difficult in large, complex scenes
* These aren't necessarily unsolvable, but a different type of solution has supplanted GANs in popularity
  * Diffusion models

---

## Diffusion

* These have many advantages of GANs
  * Except in speed. GANs generate an image in a single pass, while diffusion is iterative
* How does diffusion work?
* We once again begin with two components: an encoder and a decoder

---

## Encoder and Decoder

* The encoder takes a sample, $x$
  * Through successive encoding, it maps that back to z:
    * $x \rightarrow z_1 \rightarrow z_2 \rightarrow ... \rightarrow z_T$
* The decoder does the opposite:
  * $z_T \rightarrow z_{T-1} \rightarrow z_{T-2} \rightarrow ... \rightarrow x$

---

## Encoder Implementation

* The encoder isn't a DNN, it is just a series of steps
* Simply add Gaussian noise to x until the image is indistinguishable from Gaussian noise
  * $z_{t} = \sqrt{1 - \beta_t}z_{t-1} + \sqrt{\beta_t}\epsilon_t$
* We make sure to attenuate the current data so that we don't saturate or fall to 0

---

## Decoder Implementation

* Just train a DNN to remove the noise and restore the original image
  * So easy!

<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/UDL/Chap18/DiffusionOverview.svg" />
<br/>
<small>See the UDL text, Figure 18.1.</small>
</div>

---

## Encoding Details

* We don't actually want to spend time slowly adding noise to our image
  * But we don't have to
* Consider the first two steps where we add noise:

<p style='font-size:30pt'><math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtable><mtr><mtd columnalign="right" style="text-align: right"><msub><mi>𝐳</mi><mn>2</mn></msub></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msqrt><mrow><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub></mrow></msqrt><mrow><mo stretchy="true" form="prefix">(</mo><msqrt><mrow><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub></mrow></msqrt><mo>⋅</mo><mi>𝐱</mi><mo>+</mo><msqrt><msub><mi>β</mi><mn>1</mn></msub></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mo>+</mo><msqrt><msub><mi>β</mi><mn>2</mn></msub></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>2</mn></msub></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msqrt><mrow><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub></mrow></msqrt><mrow><mo stretchy="true" form="prefix">(</mo><msqrt><mrow><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub></mrow></msqrt><mo>⋅</mo><mi>𝐱</mi><mo>+</mo><msqrt><mrow><mn>1</mn><mo>−</mo><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow></mrow></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mo>+</mo><msqrt><msub><mi>β</mi><mn>2</mn></msub></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>2</mn></msub></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msqrt><mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow></mrow></msqrt><mo>⋅</mo><mi>𝐱</mi><mo>+</mo><msqrt><mrow><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub><mo>−</mo><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow></mrow></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>1</mn></msub><mo>+</mo><msqrt><msub><mi>β</mi><mn>2</mn></msub></msqrt><mo>⋅</mo><msub><mi>𝛜</mi><mn>2</mn></msub><mi>.</mi></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{align}
\mathbf{z}_{2} &= \sqrt{1 - \beta_{2}} \left(\sqrt{1 - \beta_{1}} \cdot \mathbf{x} + \sqrt{\beta_{1}} \cdot \mathbf{\epsilon}_{1}\right) + \sqrt{\beta_{2}} \cdot \mathbf{\epsilon}_{2} \tag{18.5} \\
&= \sqrt{1 - \beta_{2}} \left(\sqrt{1 - \beta_{1}} \cdot \mathbf{x} + \sqrt{1 - (1 - \beta_{1})} \cdot \mathbf{\epsilon}_{1}\right) + \sqrt{\beta_{2}} \cdot \mathbf{\epsilon}_{2} \\
&= \sqrt{(1 - \beta_{2})(1 - \beta_{1})} \cdot \mathbf{x} + \sqrt{1 - \beta_{2} - (1 - \beta_{2})(1 - \beta_{1})} \cdot \mathbf{\epsilon}_{1} + \sqrt{\beta_{2}} \cdot \mathbf{\epsilon}_{2}.
\end{align}</annotation></semantics></math></p>

* The last two terms are just adding i.i.d. Guassians together

---

## Encoding Details

* So we can combine the two terms:

<p style='font-size:30pt'><math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>𝐳</mi><mn>2</mn></msub><mo>=</mo><msqrt><mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow></mrow></msqrt><mo>⋅</mo><mi>𝐱</mi><mo>+</mo><msqrt><mrow><mn>1</mn><mo>−</mo><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>2</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow><mrow><mo stretchy="true" form="prefix">(</mo><mn>1</mn><mo>−</mo><msub><mi>β</mi><mn>1</mn></msub><mo stretchy="true" form="postfix">)</mo></mrow></mrow></msqrt><mo>⋅</mo><mi>𝛜</mi><mo>,</mo></mrow><annotation encoding="application/x-tex">\mathbf{z}_2 = \sqrt{(1 - \beta_2)(1 - \beta_1)} \cdot \mathbf{x} + \sqrt{1 - (1 - \beta_2)(1 - \beta_1)} \cdot \boldsymbol{\epsilon},</annotation></semantics></math></p>

* The sqrt terms can be replaced by $\alpha_t = \prod\limits_{s=1}^{t}(1 - \beta_s)$
* So we can complete encoding in a single step: $z_t = \sqrt{\alpha_t}x + \sqrt{1 - \alpha_t}\epsilon$

---

## Encoding Visualized

* Pretend the input is a single variable of some normal distribution
* Diffusion will move any input to a standard normal after enough iterations

<div class="col">
<img style="width: 60%" class="r-stretch" src="./figures/UDL/Chap18/DiffusionForward2.svg" />
<br/>
<small>See the UDL text, Figure 18.2.</small>
</div>

---

## Encoding Visualized

* Our data comes from from a complex set of variables (recall the latent vector in GANs)
* Encoding will also convert them into a normal distribution, no matter their complexity

<div class="col">
<img style="width: 60%" class="r-stretch" src="./figures/UDL/Chap18/DiffusionDensity.svg" />
<br/>
<small>See the UDL text, Figure 18.4.</small>
</div>

---

## Learning to Decode

* Decoding means predicting what the current value would look like with less noise
* Since we know $x$ during training, we can compute the probability distribution of
  * $q(z_{t−1} |z_t, x)$
* This will involve Bayes rule and some math
  * See 18.2.4 in the book, I'll omit this for the sake of brevity here

---

## Parameter Estimation

* We want to estimate the most likely $x$ given some $z_t$
* This isn't really directly tractable
  * It involves integrating over all time steps
* Instead we will use a proxy function that is always below our target, the *evidence lower bound* (ELBO)
* This involves several pages of math in the book, but it simplifies to something usable

---

## Reparameterization

* Instead of learning the function $\hat{z}_{t-1} = f[z_t, \phi]$, it can be replaced with a new function
  * Estimate the noise: $\hat{\epsilon} = g[z_t, \phi]$
  * $f[z_t, \phi] = \frac{1}{\sqrt{1 - \beta_t}}z_t - \frac{\beta_t}{\sqrt{1 - \alpha}{\sqrt{1 - \beta}}}g[z_t, \phi]$
* We are going to find it easier to estimate the noise in an image than we will to estimate the prior state itself
* After copious math, we end up with a simple loss
  * $\ell_i = \left\| \mathbf{g}_t \left[ \sqrt{\alpha_t} \mathbf{x}_i + \sqrt{1 - \alpha_t} \boldsymbol{\epsilon}, \phi_t \right] - \boldsymbol{\epsilon} \right\|^2$

---

## Loss Function

* $\ell_i = \left\| \mathbf{g}_t \left[ \sqrt{\alpha_t} \mathbf{x}_i + \sqrt{1 - \alpha_t} \boldsymbol{\epsilon}, \phi_t \right] - \boldsymbol{\epsilon} \right\|^2$
* This is a simple mean squared error
  * Given the diffused input after $t$ steps, predict the noise component

---

## Decoding

* Decoding just involves repeatedly removing noise
  * $\hat{\mathbf{z}}_{t-1} = \frac{1}{\sqrt{1 - \beta_t}} \mathbf{z}_t - \frac{\beta_t}{\sqrt{1 - \alpha_t} \sqrt{1 - \beta_t}} \mathbf{g}_t [\mathbf{z}_t, \phi_t]$
* To continue decoding, add noise to the image and repeat

---

## Easy?

* Other than the derivation, which we mostly skipped, this sounds easy
  * And is much better behaved than GANs
* But if we add a small amount of noise at a time, decoding involves a large number of steps
  * This can make diffusion slow

---

## Networks

* As a simple approach, we can use a U-Net model to estimate noise at all time steps
* Rather than training a neural network at each time step, a time embedding is added to the channel information
* Other, more sophisticated approaches use pyramidal networks that also upscale estimates

<div class="col">
<img style="width: 70%" class="r-stretch" src="./figures/UDL/Chap18/DiffusionUNet.svg" />
<br/>
<small>See the UDL text, Figure 18.4.</small>
</div>

---

## Results

* You have definitely seen results of diffusion models
* They have struggled with speed, but newer work attempts few or single step diffusion
  * Take another look at the pix2pix using diffusion [on github](https://github.com/GaParmar/img2img-turbo)

---

## Adversarial + Diffusion

* Some newer work starts with diffusion models as teachers
* A student model attempt to denoise like the teacher, and a discriminator guesses if they are real
  * [paper](https://arxiv.org/abs/2311.17042)
  * Cool website: [https://huggingface.co/stabilityai/sdxl-turbo](https://huggingface.co/stabilityai/sdxl-turbo)
  * Cool [github repository](https://github.com/Stability-AI/generative-models)

---

## Main Ideas

* Simplicity allowed people to make rapid advances with diffusion models
* And now people add more complexity back in to squeeze out better results
* Whatever techniques are used in the future, these methods are still interesting
  * GANs learn to ascribe meaning to a latent vector, or to multiple style vectors
  * Diffusion models learn to remove noise from random noise until it is a structured image
  * Both are pretty amazing