# CS 462 - Lecture 23

## Reinforcement Learning

Bernhard Firner

2026-04-16

---

## Review

* There are multiple ways to create generative models
* We talked about two of them
* GANs can generate a new sample given a latent vector
  * GANs: $G(z) \rightarrow x^*$
* Diffusion models denoise a sample, eventually reaching a plausible source
  * Diffusion: $f(z_{t}) \rightarrow z_{t-1}$

---

## Alternatives

* There are some alternatives
  * Normalizing flows and variational autoencoders
  * Chapters 16 and 17 in the book
* And some modern approaches mix them
  * Use diffusion models as a critic for adversarial training
  * Use a variational autoencoder to deduce z
  * Lots of variation

---

## Easy Loss Function

* The advantage of diffusion models is their simple loss function:
  * $\ell_i = \left\| \mathbf{g}_t \left[ \sqrt{\alpha_t} \mathbf{x}_i + \sqrt{1 - \alpha_t} \boldsymbol{\epsilon}, \phi_t \right] - \boldsymbol{\epsilon} \right\|^2$
* Given the diffused input after $t$ steps, predict the noise component
  * At each time step we estimate a mean and variance per pixel (assuming images)

---

## Decoding

* Decoding just involves repeatedly removing noise
  * $\hat{\mathbf{z}}_{t-1} = \frac{1}{\sqrt{1 - \beta_t}} \mathbf{z}_t - \frac{\beta_t}{\sqrt{1 - \alpha_t} \sqrt{1 - \beta_t}} \mathbf{g}_t [\mathbf{z}_t, \phi_t]$
* To continue decoding, add noise to the image (to prevent early collapse) and repeat

---

## Algorithm

* $\mu, \sigma^2 = D(z_t, t)$
* $z_{t-1} \leftarrow \mu + \sigma\mathcal{N}$
  * Where $\mathcal{N}$ is gaussian noise
* Notice that the denoiser, D, is going to estimate less variance as $t$ approaches 0

---

## Guidance

* Gaussian noise is logical, so let's stick with that for a moment
* We may want to control what is actually being created
* Previously, we changed the value of z
  * But here, z is created by applying noise to x many times
  * Eventually, $z_t$ converges to noise itself as t grows

---

## Gradient Guidance

* There are a few ways to shape the decoded sample, but gradients are the simplest
* If we have a classifier, we can ask it which pixels "look like" the output we want
  * This is similar to homework 4, where we examined the gradient at the image itself
  * Now we will examine the gradient associated with a class at $z_t$
  * Nudge the means of those pixels in the proper direction and (hopefully) end up with the right class

---

## Guidance Example

* $\mu, \sigma^2 = D(z_t, t)$
* $guide \leftarrow \frac{\delta\hat{y}}{\delta z_t}C(z_t)$
  * Where $C$ is a classifier (hopefully trained with noisy data)
* $\mu_{guided} = \mu + scale \times guide \times \sigma$
* $z_{t-1} \leftarrow \mu_{guided} + \sigma\mathcal{N}$
* We don't have to do things that way, and there is a hyperparemeter for scaling, but this can work

---

## Results

* Digits created with guided diffusion
* Each row was guided to a different class, 0 through 9
* The same random seed was used each time, so each column corresponds to the same starting $z$

---

## Cleanup

* Some of the outputs aren't anywhere near our desired class
  * The initial $z$ just didn't want to go there
* We could have abandonded diffusion of those images partway since we have a classifier
* In any case, we can use the classifier to choose a result with our desired class

</div>
<div class="col">
<img style="width: 10%" class="r-stretch" src="./figures/best_guided_targets.png" />
</div>
</div>

---

## Elegant And Mathematically Sound

* This sounds very logical
  * But is it?
* Gaussian noise has nice statistics, so it makes sense that we use it
* But it turns out it isn't necessary

---

## Cold Diffusion

* [Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise](https://arxiv.org/abs/2208.09392)
  * [github](https://github.com/arpitbansal297/Cold-Diffusion-Models)
  * The authors show that even static images can be decoded into arbitrary outputs
  * In fact, this can be done even with structured inputs

---

## Cold Diffusion Examples

<div class="col">
<img style="width: 70%" class="r-stretch" src="./figures/cold_diffusion-all_transform_cover.png" />
<br/>
<small>See Cold Diffusion Figure 1.</small>
</div>

---

## Implications?

* Sometimes (perhaps many times) the explanations for *why* a deep learning technique work are incomplete
* The update step for typical denoising works with gaussian noise, but not these other degredations
* The author's dug into why
  * Named the degradation step (such as adding noise) $D$, dependent upon the time step, $t$
  * Named the restoration step (such as denoising) $R$
    * $R(x_t, t) \approx x_0$

---

## Typical Algorithm

<div style="text-align: left;">
<div style="font-family: 'Times New Roman', serif; line-height: 1.6; max-width: 800px;">
  <div><strong>Input:</strong> A degraded sample <i>x<sub>t</sub></i></div>
  
  <div style="margin-left: 0;">
    <strong>for</strong> <i>s = t, t-1, ..., 1</i> <strong>do</strong>
  </div>
  
  <div style="margin-left: 2em;">
    <i>x̂<sub>0</sub></i> ← <i>R(x<sub>s</sub>, s)</i>
  </div>
  
  <div style="margin-left: 2em;">
    <i>x<sub>s-1</sub></i> = <i>D(x̂<sub>0</sub>, s-1)</i>
  </div>
  
  <div style="margin-left: 0;">
    <strong>end for</strong>
  </div>
  
  <div><strong>Return:</strong> <i>x<sub>0</sub></i></div>
</div>
</div>

---

## Operation

* Obviously we want $R(D(x_0, t)) = x_0$, meaning that $R$ perfectly reversed $D$
* What happens if there are errors?
  * The function, $R$, corrects random gaussian errors anyway
  * So if there is some residual error, it doesn't introduce something that R cannot correct
* That equation does not work with the degradations in the Cold Diffusion paper though

---

## Cold Vs Hot

* Plenty of algorithms use a metaphorical temperature to describe their operation
* In this case, the noise at $t=600$ is "high", so we would consider that a "hot" image
* As the image is denoised, we "cool" it, removing noise, until reaching a final state
* But what if the degredation was completely deterministic, in effect "cold" and lacking any additional entropy?

---

## Updated Algorithm

<div style="text-align: left;">
<div style="font-family: 'Times New Roman', serif; line-height: 1.6; max-width: 800px; color: #1a1a1a;">
  <div><strong>Input:</strong> A degraded sample <i>x<sub>t</sub></i></div>
  
  <div style="margin-top: 5px;">
    <strong>for</strong> <i>s = t, t-1, ..., 1</i> <strong>do</strong>
  </div>
  
  <div style="margin-left: 2em;">
    <i>x̂<sub>0</sub></i> ← <i>R(x<sub>s</sub>, s)</i>
  </div>
  
  <div style="margin-left: 2em;">
    <i>x<sub>s-1</sub></i> = <i>x<sub>s</sub></i> − <i>D(x̂<sub>0</sub>, s)</i> + <i>D(x̂<sub>0</sub>, s-1)</i>
  </div>
  
  <div style="margin-top: 5px;">
    <strong>end for</strong>
  </div>
  
  <div style="margin-top: 5px;"><strong>Return:</strong> <i>x<sub>0</sub></i></div>
</div>
</div>

---

## Discussion

* If the restoration function is perfect, the second algorithm offers no advantage
  * What if $R$ is imperfect?
* Let's say $D$ introduced a change that scales with the time step
  * $D(x,s) \approx x + s\cdot \mathcal{e}$
* Let's expand the second algorithm
<div style="font-family: 'Times New Roman', serif; line-height: 1.6; max-width: 800px; color: #1a1a1a;">
  <div style="margin-left: 2em;">
    <i>x<sub>s-1</sub></i> = <i>x<sub>s</sub></i> − <i>D(x̂<sub>0</sub>, s)</i> + <i>D(x̂<sub>0</sub>, s-1)</i>
  </div>
</div>

---

<p><math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mtable><mtr><mtd columnalign="right" style="text-align: right"><msub><mi>x</mi><mrow><mi>s</mi><mo>−</mo><mn>1</mn></mrow></msub></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msub><mi>x</mi><mi>s</mi></msub><mo>−</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>+</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>,</mo><mi>s</mi><mo>−</mo><mn>1</mn><mo stretchy="true" form="postfix">)</mo></mrow></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mn>0</mn></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>−</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>+</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>,</mo><mi>s</mi><mo>−</mo><mn>1</mn><mo stretchy="true" form="postfix">)</mo></mrow></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msub><mi>x</mi><mn>0</mn></msub><mo>+</mo><mi>s</mi><mo>⋅</mo><mi>e</mi><mo>−</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>−</mo><mi>s</mi><mo>⋅</mo><mi>e</mi><mo>+</mo><mi>R</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mi>s</mi></msub><mo>,</mo><mi>s</mi><mo stretchy="true" form="postfix">)</mo></mrow><mo>+</mo><mrow><mo stretchy="true" form="prefix">(</mo><mi>s</mi><mo>−</mo><mn>1</mn><mo stretchy="true" form="postfix">)</mo></mrow><mo>⋅</mo><mi>e</mi></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><msub><mi>x</mi><mn>0</mn></msub><mo>+</mo><mrow><mo stretchy="true" form="prefix">(</mo><mi>s</mi><mo>−</mo><mn>1</mn><mo stretchy="true" form="postfix">)</mo></mrow><mo>⋅</mo><mi>e</mi></mtd></mtr><mtr><mtd columnalign="right" style="text-align: right"></mtd><mtd columnalign="left" style="text-align: left"><mo>=</mo><mi>D</mi><mrow><mo stretchy="true" form="prefix">(</mo><msub><mi>x</mi><mn>0</mn></msub><mo>,</mo><mi>s</mi><mo>−</mo><mn>1</mn><mo stretchy="true" form="postfix">)</mo></mrow></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{aligned}
x_{s-1} &= x_s - D(R(x_s, s), s) + D(R(x_s, s), s-1) \\
&= D(x_0, s) - D(R(x_s, s), s) + D(R(x_s, s), s-1) \\
&= x_0 + s \cdot e - R(x_s, s) - s \cdot e + R(x_s, s) + (s-1) \cdot e \\
&= x_0 + (s-1) \cdot e \\
&= D(x_0, s-1)
\end{aligned}</annotation></semantics></math></p>

* Notice that the restored value from $R$ cancels
* Also notice that $D(x_0, s-1)$ will just be $x_0$ if $s=1$
  * So this will always restore the original image, regardless of $R$

---

## Freedom of R

* Using that equation, the [Cold Diffusion](https://arxiv.org/abs/2208.09392) authors go on to invert such functions as:
  * Blurring
  * Down-sampling to 2x2 pixels
  * Inpainting
  * Snowification (a type of non-gaussian noise)
  * Animorphosis, where an image is transformed into an animal face from AFHQ and back
* The last one shows that the transformation is arbitrary
  * Results aren't necessarily *good*, but they do demonstrate that any arbitrary starting point is okay

---

## Key Points

* The field is changing rapidly, so focus on some key points
* First, latent vectors and spaces have meaning
  * z maps into the range of possible X values
  * If trained poorly, there is no structure to z and different features are entangled

</div>
<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/z_mapping.svg" />
</div>
</div>

---

## Key Points

* There are multiple ways to find $z$ values that map into specific properties of $X$
* Just think of the incredible improvements we've seen in image generation in recent years
* Common problems: mode collapse and entanglement still show up
  * Less noticeable, but still present

</div>
<div class="col">
<img style="width: 80%" class="r-stretch" src="./figures/z_mapping.svg" />
</div>
</div>

---

## DL Topics

* At this point we have covered many uses of deep learning
* Are we missing any important topics?
  * Yes!
* All of our examples are not interactive
  * No interaction with an environment, human, etc

---

## Loss and the Real World

* Our loss functions have been optimizing a DNN output given a specific input
* If the DNN is being used in a multi-step process, this can lead to problems
* For example, let's say you train a DNN to predict steering wheel angles given an image
  * Over time, the loss is very small, so predictions are good
  * Can the model drive?

---

## Udacity Example

* This describes the Udacity self-driving car dataset
* Imagine that the DNN is slightly biased left at all times
  * What happens?
  * It slowly accumulates and bad angle and drives off the road
* Even with a very low loss, a DNN trained this way cannot drive

---

## Why?

* We can blame two things
* First, the Udacity images are all taken from real driving
  * So none of them show a vehicle that is pointed in the wrong direction
* Second, the DNN is not "planning ahead" and thinking about where it will end up
* We could choose to fix either of those

---

## Fixing the Data

* We could fix the data with augmentations
* For example, [PilotNet](https://arxiv.org/abs/2010.08776) augmented real-world data to produce images showing the car in bad positions

---

## Fixing Planning

* This is harder to do
* The loss function does not account for the value of an action over multiple time steps
* This is the domain of reinforcement learning
  * Chapter 19 in the UDL book
* It generally requires a simulator (or a willingness to break robots)
  * But the augmentations being done in PilotNet could be sufficient for simulation

---

## RL for LLMs

* Let's take an example that you've likely seen
* A transformer can predict which words should show up in response to other words
  * This is basically our LLMs
* However, the word predictions are probabilities, not certain answers
  * As with autonomous driving, some choices may drive the LLM's future text away from the desired trajectory

---

## Human Feedback

* The early solution to this was to have humans rate the responses of LLMs
* In [Learning to Summarize with Human Feedback](https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html) the authors describe how human feedback can be used
  * Humans select between different summaries, building a dataset of preferences
  * A model is then trained to predict which outputs are preferred
  * This is then used to provide feedback to the LLM, altering its outputs to be preferred by humans

---

## Inhuman Feedback

* New work removes the human from the loop
  * See [RLAIF vs RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://arxiv.org/abs/2309.00267)
* But how does this feedback work?
  * We can't change the probabilities of the language model itself
  * Instead, we need to create a *policy* that selects words while still using the language model to create the word suggestions

---

## Reinforcement Learning

* Learning a policy is the domain of reinforcement learning
  * It is 25 pages of the UDL book, but is its own field with its own text books
* So we will merely be covering some general topics
  * Hopefully enough to demystify RL and give you a starting point if you want to learn more

---

## Terms

* RL is usually described as the problem of optimizing the *rewards* of an *agent* as it interacts with an *environment*
* For the LLM example, the agent is creating text, the environment is an interactive web page or phone app, and the reward is whether or not a human finds the response useful
* Rewards themselves tend to be vague, and may not have explicit values
  * But as long as different outcomes have a ranking of preference or indifference, RL can work

---

## Values

* We assign a *value* (sometimes called utility) to outcomes
  * This can be arbitrary, as long as more preferred outcomes have a larger value than less preferred outcomes
* Most environments are episodic, meaning that there is a terminal state
  * For example, when the user stops chatting with the LLM
* There may be a reward per action
* And a final gain for the quality of the terminal state

---

## Rewards

* The rewards and final *gain* are associated with the actions taken
  * Future rewards are discounted by $\gamma$, which represents some uncertainty about future rewards
  * $G_t \leftarrow R_{t+1} + \gamma G_{t+1}$
* If we expand that equation, we get $G_t = \sum\limits_{k=0}^T\gamma^kr_{t+k+1}$

---

## Policies and Learning

* A policy, $\pi$, assigns action probabilities given the current state: $\pi(s) = p(a|s)$
  * An optimal policy will try to maximize the future reward
* We can attempt to learn the policy directly
* Or we can learn the rewards of an action given the current state
* The second is simpler, but won't work well if actions are continuous values

---

## Q Learning

* The first option, learning an estimate of rewards given the current state an an action, is called Q learning
  * The value estimating function is $Q(a, s)$
* For simple problems Q is a lookup table
* But if the state space is continuous, it helps to represent Q using a neural network
  * This is called a Deep Q Network, or DQN

---

## Examples

* With all of that introduction out of the way, let's do an example
* We'll use environments from Farama-Foundation [Gymnasium](https://github.com/Farama-Foundation/Gymnasium)
  * Has lots of environments already set up for learning
* Install with `pip3 install gymnasium[all]`
* We'll go into Q-learning details next class, but let's look at some examples

---

## Cliff Walking

* [Cliff Walking](https://gymnasium.farama.org/environments/toy_text/cliff_walking/)
* This is a problem easily solved without a DNN
* Just a "hello world"
* Every step has a reward of -1, but falling off of the cliff has a reward of -100

---

## Cliff Walking Policy

* There are four actions in each position, so the DNN is being asked to predict a reward for each one
* Our optimal policy chooses the "best" action each time
  * $\pi(S_t) \leftarrow \underset{a}{argmax} Q(S_t, a)$

---

## Q-Learning

* Q (our dnn) begins with random initialization, so it needs to learn as we explore
* In tabular learning (where we just record values in a table), updates look like this:
  * $Q(S_t,A) \leftarrow Q(S_t,A) + \alpha[R + \gamma \underset{a}{max}Q(S_{t+1},a) - Q(S_t,A)]$
  * $\alpha$ is the learning rate, $\gamma$ is the discount of future rewards

---

## Q Values

* We'll get into the details next class
* To explore, we take a random action with probability $\epsilon$, otherwise we follow our policy
* Over time, we learn that going down from the cliff is bad

</div>
<div class="col">
<img style="width: 70%" class="r-stretch" src="./figures/cliff-walking-dqn-example.gif" />
</div>
</div>

---

## Why Use a DNN?

* In cliff walking, using a DNN doesn't offer any advantage over a lookup table with probabilities
* Deep learning is useful if the state itself is continuous
  * Now Q will be similar to any other value estimation DNN
  * Actions are categorical predictions
* And when the actions are continuous
  * Now the actions become parameters estimates of a distribution

---

## Mountain Car

* [Mountain Car](https://gymnasium.farama.org/environments/classic_control/mountain_car/)
* This is the discrete action version, where control is -1 (go backwards), 0, or 1 (go forwards)
* The continuous version, with a value from -1 to 1, is actually much more difficult
* We'll go into more detail next time

<!--
RLAIF vs RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
https://arxiv.org/abs/2309.00267

RL in LLMs
Amazon: https://aws.amazon.com/blogs/machine-learning/fine-tune-large-language-models-with-reinforcement-learning-from-human-or-ai-feedback/

So RL is important. How does it work?

-->

<!--

Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
https://arxiv.org/abs/2208.09392
https://github.com/arpitbansal297/Cold-Diffusion-Models

-->