<!--
Abstract:

CS 461
Introduction to Deep Learning
Lecture 11
-->

# CS 461 - Lecture 21

## Machine Learning Principles

### Alexnet

Bernhard Firner

2025-11-17

---

## Alexnet

* [Original Paper](https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html)

---

## Alexnet and Imagenet

* Alexnet (named for author Alex Krizhevsky)
  * Began the current hype
* Huge jump in performance; 15.3% top-5 test error vs 26.2% for second best
* Dataset was actually useful
  * Imagenet was made up of large images from the internet
  * So the DNN was immediately useful

---

## Dataset

* Let's pause for a moment to discuss why ImageNet was so hard
* First, it's a crappy dataset
  * Classes don't make any sense
    * Toilet paper and toilet seat, really?
    * And they overlap too!

---

## Examples

What image class is this?

---

## Examples

It's a water snake!

Duck? What duck?

---

## Soccer Ball!

Dog? Don't be silly.

---

## Soccer Ball? Bird?

It's a hornbill.

---

## Fiddler Crab

---

## Lobster

---

## Jellyfish?

---

## Which one?

Whale? Boat? Person?

---

## Correlation Issues

* Some classes are frequently coincident
  * One dog breed with another
  * Hunting dogs with birds
  * Toilet paper with toilet seats

---

## Structural Issues

* Some make little sense
  * Why should cooked food or a toy of something be the same class?
* And there are different classes for nearly indistinguishable things
  * Many, many dog breeds
* Some images are thumbnails, others are full res digital camera images

---

## Top-5

* It's impossible to get the correct answer
* So classifiers make predictions and the top 5 are chosen
  * If the model has the correct class in that top 5, it is correct

---

## Why?

* Q: Why use this?
* A: It is huge. Over 14 million images.

---

## Problem: Overfitting

* I've stressed that overfitting isn't a thing **as long as your training data and testing data match**
* The labels in ImageNet are, at best, ambiguous
  * And some are wrong or bizarre
    * Many images of soggy toilet paper and cats pooping while standing on toilet seats
    * I cannot stress how bizarre this dataset is
* But where else can we find so much data?

---

## Best Practices

* [Limits on Learning Machine Accuracy Imposed by Data Quality](https://proceedings.neurips.cc/paper_files/paper/1994/file/1e056d2b0ebd5c878c550da6ac5d3724-Paper.pdf)
  * Paper from 1994
* Describes problems trying to get accuracy beyond the quality of your dataset

---

## Capacity

* Let's call the capacity of our model *h*
* We train with $l$ samples
* If $h << l$ the training error is high
  * This is underfitting
* If $h >> l$ there should be no training error
  * Whether we should call that overfitting depends upon the quality of our dataset

---

## Overfitting

* Most datasets have "noise"
  * Errors in labels
  * Impossible to learn examples
* We add capacity, and the model learn special cases
  * Dogs are always dogs unless they have a soccer ball
  * Soccer balls are soccer balls unless held by a bird
* The capacity that is too much is $h*$ in the figure

</div>
<div class="col">

</div>
</div>

---

## Error Vs Capacity

* If we remain below $h*$, adding more training data should eventually cause our training and testing errors to meet
  * Trivially, as we add every single possible input to the training set
  * But should happen before then

---

## Error Vs Capacity

* The error that we're left with on our training set is usually something we have to accept
  * Pushing the model harder will only result in learning noise rather than signal

---

## Back to Alexnet

* Different classes have different level of mislabelling or ambiguity
* Decreasing capacity so those aren't learned will cause us to fail on other examples
* So what can we do?
  * Regularize like **crazy**

---

## Intro to Alexnet

> "In the end, the network’s size is limited mainly by the amount of memory available on current GPUs
> and by the amount of training time that we are willing to tolerate. Our network takes between five
> and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results
> can be improved simply by waiting for faster GPUs and bigger datasets to become available."

---

## Bigger

* Alexnet is bigger than LeNet
  * Bigger convolutions
    * 11x11, 5x5, three 3x3 kernels
* Each one has enough padding that they operate on features near the edge
  * LeNet only padded on the input image

---

## Pooling

* Bigger pooling
  * Pooling now covers overlapping pixels
  * 3x3 pooling, stride 2
  * Take the maximum value in the 3x3 area
* Smoother than a 2x2 stride 2 pool
  * Since adjacent weights are shared
* This is a kind of regularization

---

## ReLU

* ReLU is used in place of tanh
  * Because it learns faster
* No discussions of quality of fit

</div>
<div class="col">

</div>
</div>

---

## ReLU Vs Tanh

</div>
<div class="col">

</div>
</div>

Maybe, With enough data, the ReLU boundary will also become smooth.

---

## Local Response Normalization

* Tanh inputs need to be kept in a narrow range to avoid saturating
* ReLU won't saturate, but some inputs must be positive
  * We don't want to subtract by the mean, but do want to rescale outputs
* The pixels of adjacent feature maps are divided by a function of the sum of the squares of their neighbors

---

## Local Response Normalization

* It is claimed that this "implements a form of lateral inhibition inspired by the type found in real neurons"
  * The kind of cool-sounding nonsense claim that we love in modern NN papers
* $LRN(x_i) = \frac{x_i}{(2 + 10^{-4}\sum_{j=i-2}^{i+2}x^2)^{0.75}}$
* The four constants (2, $10^{-4}$, 0.75, and the number of neighbors) come from testing on a validation set

---

## Structure

* 11x11 and 5x5 conv layers are followed by response normalization and max pooling
* The first 3x3 conv is followed by max pooling
* Three linear layers for classification
* ReLU is applied to the output of every convolution and linear layer

---

## Structure

---

## More Regularization

* Training set is still noisy, so this is required
* Two techniques
  * Data augmentation
  * Dropout

---

## Data Augmentation 1

* During training
  * Extract (or scale) 224x224 random patches
  * Randomization includes translations and horizontal reflections
* During testing
  * Extract five 224x224 patches (four corners and a center patch)
  * Add in five reflections
  * Average predictions over all 10 patches

---

## Data Augmentation 2

* During training
  * Perform PCA on all RGB pixel values
  * Add multiples of the principal compoments
    * eigenvalue * random gaussian with 0 mean and 0.1 stddev
* Nothing done during testing

---

## Dropout

* We want an ensemble of models as regularization
  * But training multiple models is too data and time intense!
* So instead, we'll use dropout
* Randomly zero the outputs of hidden neurons with some probability
  * That way, each forward-backward pass only uses a subset of the network

---

## What is Dropout?

* During training, randomly ignore some neurons
* For example, given neurons a, b, c, and d, drop half at each training step:

1. $f(x) = a + d$ 
1. $f(x) = a + c$ 
1. $f(x) = b + c$ 
1. $f(x) = a + c$

* When training is done, use them all 
* Now there are four numbers instead of two, so divide by half to preserve average outputs magnitude

$f(x) = (a + b + c + d)/2$

---

## Realistic Inputs

Suppose that your training data has two signals that are equal to one another.

```python [|7-13|15-28|30-31|35-42|46-52]
import random
import torch

# For better repeatability
torch.random.manual_seed(0)

# Imagine that the inputs represent features in an image
net = torch.nn.Sequential(
        torch.nn.Linear(2, 100),
        torch.nn.ReLU(),
        torch.nn.Linear(100, 100),
        torch.nn.ReLU(),
        torch.nn.Linear(100, 1))

# Our training set.
def make_input_outputs(size):
    with torch.no_grad():
        # Make a batch of inputs that is just the same pairs of numbers
        inputs = torch.empty([size, 1]).uniform_(0, 1).repeat(1, 2)
        outputs = inputs[:,0].view((size, 1))
        for idx in range(size):
            # 1/1000 chance that a signal is missing.
            if random.random() < 0.001:
                inputs[idx,0] = 0
            if random.random() < 0.001:
                inputs[idx,1] = 0

return inputs, outputs

optimizer = torch.optim.SGD(net.parameters(), lr=0.004, momentum=0.05, weight_decay=0.01)
loss_fn = torch.nn.MSELoss(reduction='sum')

net.train()

# Train for a long time.
for step in range(10000):
    x_inputs, y_targets = make_input_outputs(64)
    optimizer.zero_grad()
    output = net.forward(x_inputs)
    loss = loss_fn(output, y_targets)
    loss.backward()
    optimizer.step()

net.eval()

# Probe the network to test how it learned.
print("input a, input b, output")
for a in range(101):
    for b in range(101):
        probe = torch.tensor([a/100, b/100])
        output = net.forward(probe)
        print(f"{a/100}, {b/100}, {output.item()}")
```

---

## Biased Outputs

![](./figures/bad-signals.png)

* Cares more about input 1 than input 2. Why?
* It arbitrarily used 1 and not 2 because no loss prevented that result.

---

## With Dropout

```python [|11,14]
import random
import torch

# For better repeatability
torch.random.manual_seed(0)

# Imagine that the inputs represent features in an image
net = torch.nn.Sequential(
        torch.nn.Linear(2, 100),
        torch.nn.ReLU(),
        torch.nn.Dropout(0.5),
        torch.nn.Linear(100, 100),
        torch.nn.ReLU(),
        torch.nn.Dropout(0.5),
        torch.nn.Linear(100, 1))

return inputs, outputs

optimizer = torch.optim.SGD(net.parameters(), lr=0.004, momentum=0.05, weight_decay=0.01)
loss_fn = torch.nn.MSELoss(reduction='sum')

net.train()

net.eval()

---

## Unbiased Outputs

![](./figures/bad-signals-dropout.png)

* Much less biased

---

## Is that better?

* If two signals carry equal meaning, then the network should treat them similarly
* Dropout is like training a "subset" of the network at every pass
* Since the bias of the original network was arbitrary, this averages over many biased models, produced one with less bias

---

## Dropout in Alexnet

* A dropout probability of 0.5 was used during training
* During evaluation, dropout is turned off and outputs are multiplied by 0.5
  * Otherwise, the magnitude of the output would be too high
* This is applied after the 11x11 and 5x5 convolution layers
* Roughly doubles training time

---

## Other Training Parameters

* Trained with SGD and batch size of 128
* Weight decay is also used (0.0005)
  * Remember that this is our old friend, the $\mathcal{l}_2$ norm
  * But a large weight decay slightly degrades performance, so it is not a strong regularizer

---

## Note on Size

* Many architectures can achieve the same performance
* And we know now that there are smaller networks
* [https://arxiv.org/abs/1602.07360](https://arxiv.org/abs/1602.07360)
  * 50x fewer parameters than AlexNet, same accuracy

---

## Results

* 15.3% top-5 test error vs 26.2% for second best
  * And trained on a GPU!
* Around 65% top-1 accuracy
  * Current DNNs pretrained on other data are in the low 90% range
    * Constrained by the quality of the labels

---

## Other Results

* There are other results
  * We actually care about their observations more than some score

---

## Example Classifications

---

## Not Probabilities

* The "top 5" scores aren't really probabilities
* Why? Because they don't have statistical meaning
  * Agaric is a type of mushroom
  * Grilles are parts of a car
* The network is actually being asked to both identify image classes and guess how a human labelled them

---

## Useless?

* So if the labels aren't useful, what good is this?
  * Why all the hype?

---

## Feature Vectors

* Cut off the linear layer and look at the convolution features
  * The vector is incredibly useful
* Several uses
  * Train new classifiers using these features
    * Not just DNNs! SVMs work too!
  * Cluster images

---

## Example Similarity

* Column 1 from the test set
* Other columns have nearest feature vectors

---

## Observations

* Notice that the poses are not critical
  * Which means that the feature vector is encoding *semantics*
* Are the features learned from this task *universal*?
  * This is a big deal!

---

## Feature Compression

* In fact, future work showed that Alexnet features were applicable to a wide range of problem classes
  * This is a surprising result!
* It also unlocks other techniques in surprising ways

---

## Example Use Case
* SVMs work well with small amounts of data
  * But they don't work well (or as well as a large DNN) on high-dimensional image data
* However, large DNNs require a great deal of labelling
* Can we use a DNN feature vector to train an SVM on novel classes with a small number of labels?
  * Yes!

---

## Autoencoders

* Take that a step farther
  * If the labels don't matter, we can train a DNN on *anything*
  * Then use it as a feature extractor
* We won't need high quality labels
  * Maybe we won't even need labels at all

---

## Next Class

* Talk about improvements to AlexNet
* Also talk about using feature vectors as encodings
* And the quiz

<!--

Takeaways: abuse regularizers
No idea of anything in the dataset

Interesting parts:
1. example image results
2. six training images with feature vectors that have the smallest euclidean distances
  2. This implies something about the feature space
  2. This is a 4096D space, far smaller than the original 224x224x3
-->