# CS 462 - Lecture 20

Self-Supervised Learning

Bernhard Firner

2026-04-09

---

## Self-Supervised Learning

* Last time we reviewed a bonanza of interesting papers
* [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377) from 2021
* [Emerging Properties in Self-Supervised Vision Transformers](https://openaccess.thecvf.com/content/ICCV2021/html/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper) from 2021
* [Concept Generalization in Visual Representation Learning](https://openaccess.thecvf.com/content/ICCV2021/html/Sariyildiz_Concept_Generalization_in_Visual_Representation_Learning_ICCV_2021_paper.html) from 2021
* [ConvNeXt V2](https://openaccess.thecvf.com/content/CVPR2023/html/Woo_ConvNeXt_V2_Co-Designing_and_Scaling_ConvNets_With_Masked_Autoencoders_CVPR_2023_paper.html) from 2023

---

## History

* Some of these advances in self-supervised learning were prompted by transformers
  * Self-supervised learning was important for langauge models, but hadn't been as useful with images
* But transformers made masking so easy, so eventually people got masking working

<div class="col">
<img style="width: 50%" class="r-stretch" src="./figures/MAE_figure1.png" />
<br/>
<small> See Figure 1 in <a href="https://arxiv.org/abs/2111.06377">Masked Autoencoders Are Scalable Vision Learners</a></small>
</div>

---

## Magic!

* Examples from [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377)

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./figures/MAE_imagenet_decode.png" />
<br/>
<small>ImageNet validation image</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./figures/MAE_coco_decode.png" />
<br/>
<small>Coco validation image</small>
</div>
</div>

---

## Good?

* The results are cool, but were they good?
  * Yes!
* Look at the clean feature maps from [Emerging Properties in Self-Supervised Vision Transformers](https://openaccess.thecvf.com/content/ICCV2021/html/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper)

<div class="col">
<img style="width: 45%" class="r-stretch" src="./figures/EmergingProperties_figure1.png" />
<br/>
<small>See Figure 1 of the paper. Images show attention payed to 8x8 patches by the [CLS] token by the multi-headed attention in a self-supervised transformer's last layer.</small>
</div>

---

## Feature Maps

* Why do we care about feature maps? Do they really indicate model quality?
* You've just been assigned a homework on feature saliency
  * So let's soujorn down this side topic for a few seconds

---

## Espresso Saliency Maps

* Saliency maps are similar to the feature masking from the paper we discussed
* They show what the model is "looking at", but are unrelated to any particular class

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_no_class_original.png" />
<br/>
<small>The original (rescaled) espresso maker.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_no_class_saliency.png" />
<br/>
<small>The three highest magnitude saliency maps.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_no_class_combined.png" />
<br/>
<small>The saliency overlayed on the image.</small>
</div>
</div>

---

## Espresso Saliency Maps

* Let's look at the pixels of the image that the model is associating with particular classes
* We'll begin with the true class
  * No surprises here

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_maker_class_original.png" />
<br/>
<small>The original (rescaled) espresso maker.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_maker_class_saliency.png" />
<br/>
<small>The pixels strongly correlated with the "espresso maker" class, in red.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_maker_class_combined.png" />
<br/>
<small>Saliency overlayed on the image.</small>
</div>
</div>

---

## Espresso Saliency Maps

* What if we look at the "espresso" class?
  * There are two of them in the picture

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_class_original.png" />
<br/>
<small>The original (rescaled) espresso maker.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_class_saliency.png" />
<br/>
<small>The pixels strongly correlated with the "espresso" class, in red.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_espresso_class_combined.png" />
<br/>
<small>Saliency overlayed on the image.</small>
</div>
</div>

---

## Espresso Saliency Maps

* What if we look for something that isn't there?
  * How about a banana?

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_banana_class_original.png" />
<br/>
<small>The original (rescaled) espresso maker.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_banana_class_saliency.png" />
<br/>
<small>The pixels strongly correlated with the "banana" class, in red.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/convnext_espresso_maker_banana_class_combined.png" />
<br/>
<small>Saliency overlayed on the image.</small>
</div>
</div>

---

## Quality

* So this confirms that the model isn't just looking at random stuff.
  * Would any model actually do that?
  * Of course! Let's look at AlexNet
* Here are three saliency maps. Which classes go with the three pictures?

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_espresso_class_saliency.png" />
<br/>
<small>Door number 1.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_banana_class_saliency.png" />
<br/>
<small>Door number 2.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_espresso_maker_class_saliency.png" />
<br/>
<small>Door number 3.</small>
</div>
</div>

---

## The Reveal

* Saliency and features tell us *a lot* about a model
  * Quality, what it is looking at

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_espresso_class_combined.png" />
<br/>
<small>Espresso saliency.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_banana_class_combined.png" />
<br/>
<small>Banana saliency.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/alexnet_espresso_maker_espresso_maker_class_combined.png" />
<br/>
<small>Espresso maker saliency.</small>
</div>
</div>

---

## Overfitting

* With the swin transformer, we can even see some kind of strange pixel artifact on the left
  * It doesn't show up for banana or espresso classes, but it must be correlated with something
  * There is a good chance that the model learned a very specific, nonsense rule; classic overfitting

<div class="container">
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/swin_espresso_maker_no_class_original.png" />
<br/>
<small>The original (rescaled) espresso maker.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/swin_espresso_maker_no_class_saliency.png" />
<br/>
<small>The strongest features.</small>
</div>
<div class="col">
<img style="width: 75%" class="r-stretch" src="./homeworks/hw04/hw04_examples/swin_espresso_maker_no_class_combined.png" />
<br/>
<small>Saliency overlayed on the image.</small>
</div>
</div>

---

## More Evidence

* There more, if we weren't convinced by feature maps and saliency
* [Concept Generalization in Visual Representation Learning](https://openaccess.thecvf.com/content/ICCV2021/html/Sariyildiz_Concept_Generalization_in_Visual_Representation_Learning_ICCV_2021_paper.html) compared a large number of diverse networks
  * They created a dataset of unseen classes, and ranked them by their distance to the original training data
* The models were worse on unseen data, but self-supervised models retained the most, and recovered the quickest with a small amount of new training data

---

## Self-Supervision

* So how do we get such good self-supervised results?
  * One approach is masking and reconstruction, similar to text models
* Another approach is contrastive learning
* This is when we ask two networks to produce similar results for similar inputs
  * Conceptually similar to masking, but the training approach is very different

---

## Contrastive Learning

* There are many versions of this, but let's begin by talking about a simple one
  * [A Simple Framework for Contrastive Learning of Visual Representations](https://arxiv.org/abs/2002.05709)
* Contrastive learning means we'll be comparing two things
  * In this case, we will take an input, $x$, and applies two different transformations
  * Let's call thoise samples $\tilde{x}_i$ and $\tilde{x}_j$

---

## Transformations?

* The transformation itself has to be somewhat information preserving
  * Random cropping and resize, color distortions, or Gaussian blurs
* Afterwards $\tilde{x}_i$ and $\tilde{x}_j$ have different pixels
  * although they still share the same content

---

## Embedding and Agreement

* Features are extracted from $\tilde{x}_i$ and $\tilde{x}_j$, such as through a convnet
  * Let's call those outputs $h_i$ and $h_j$
* We then project those, such as with a small linear network, to create $z_i$ and $z_j$
* Apply a contrastive loss to the two $z$ terms
  * Given a set $\{\tilde{x}_k\}$, predict $j$ given $\tilde{x}_i$

---

## Loss Function

* The comparison is a scaled dot product similarity of the projected vectors, $z$
  * $similarity(u, v) = \frac{u^Tv}{||u||~||v||}$
  * $s_{u,v}$ for short
* The full loss is a softmax run over all similarities within the batch
  * $\ell(i, j) = -log\frac{exp(s_{i,j}/\tau)}{\sum\limits_{k=1}^{2N}\mathbb{1}_{[k\neq i]}exp({s_i,k} / \tau)}$
* $\tau$ is used as a temperature (hyperparameter) to control the rate of change

---

## Analogy

* The set is going to come from a random training batch
* You can think of this as being asked to pick someone out of a lineup
  * They may be the same person, but the images of them are different
  * In order to make a match, you need to know something fundamental about human faces
* The NN version is a bit harder; you may be asked to match to people given an image of a foot and a hand

---

## Difficult Problems

* This is somewhat similar to the masking problem
  * It is very hard
  * The huge variety of augmentations means we don't need to worry about overfitting

---

## Result

* In the end, the encoder network is kept
  * And now you have a set of features that are (maybe) invariant to many augmentations!
  * The only way that is possible is if the network understands the fundamental structure of its subjects

---

## More Complications

* The excellent feature maps we looked at before have a more advanced approach
  * [Emerging Properties in Self-Supervised Vision Transformers](https://openaccess.thecvf.com/content/ICCV2021/html/Caron_Emerging_Properties_in_Self-Supervised_Vision_Transformers_ICCV_2021_paper) suggested a technique named *DINO*: **Di**stillation with **no** labels
* The concept is almost the same, but feature extraction is done with a *student* and a *teacher*

---

## Student and Teachert

* The student network is updated to have outputs more similar to the teach
* The teacher network uses an exponential moving average of the student network weights
  * Called momentum training
* The momentum training comes from [Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning](https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html)
* But why would we do this?

---

## Motivation

* Momentum training removes the need for negative pairs (the non-matching samples in the batch)
  * But they still optimize to maximize the similarity between two augmented pairs
* Without negative examples, previous work had seen *collapse*
  * This is when a neural network gets stuck in a particular state
  * For example, if the network just output features of all 1s, all matching pairs would have high similarity

---

## EMA Weights

* Fundamentally, if a gradient existed for both the student and the teacher then learning cannot be stable
* Now, the models are different, so they cannot collapse to an easy solution
  * But they are similar, since one is being updated with the exponential moving average of the other's weights
  * $\theta_{teacher} \leftarrow \tau \theta_{teacher} (1 - \tau)\theta_{student}$

---

## More Topics

* We've been talking about unsupervised pretraining of general-purpose feature extractors
  * But there is more to unsupervised learning than that
  * So let's give a couple of examples

---

## Homography

* Let's say that you want to estimate relative changes in pose from one image to another
  * Perhaps you are a robot or a delivery drone
* Accurate motion estimate are generally difficult, and involve quite a bit of filtering and estimation

---

## Learning Homography

* A neural network could learn homography
  * For example, if you use some high-quality sensors, you could ground truth the rotation and translation of the camera between frames
  * Now feed the images to a DNN and ask it to predict the homogrophy matrix
* That sounds good, but can you really collect a large dataset like that?

---

## Unsupervised Homography

* [Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model](https://arxiv.org/abs/1709.03966) had a different approach
  * Train a network to predict the homography without any labels

<div class="col">
<img style="width: 50%" class="r-stretch" src="./figures/UnsupervisedDeepHomography_fig1.png" />
<br/>
<small> See Figure 1</small>
</div>

---

## Formulation

* A point in space, $(x,y,z)^T$ is equivalent to $(x/z,y/z,1)^T = (u,v,1)^T$
* $x = (u,v,1)^T$ and $x' = (u',v',1)^T$ are two matching points on different images
* A homography maps between those image points: $x' = Hx$

<p><math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mo stretchy="true" form="prefix">[</mo><mtable><mtr><mtd columnalign="center" style="text-align: center"><mi>u</mi><mi>′</mi></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mi>v</mi><mi>′</mi></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mn>1</mn></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mspace width="1.0em"></mspace></mtd></mtr></mtable><mo stretchy="true" form="postfix">]</mo></mrow><mo>=</mo><mrow><mo stretchy="true" form="prefix">[</mo><mtable><mtr><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>11</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>12</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>13</mn></msub></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>21</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>22</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>23</mn></msub></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>31</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>32</mn></msub></mtd><mtd columnalign="center" style="text-align: center"><msub><mi>h</mi><mn>33</mn></msub></mtd></mtr></mtable><mo stretchy="true" form="postfix">]</mo></mrow><mrow><mo stretchy="true" form="prefix">[</mo><mtable><mtr><mtd columnalign="center" style="text-align: center"><mi>u</mi></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mi>v</mi></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mn>1</mn></mtd></mtr><mtr><mtd columnalign="center" style="text-align: center"><mspace width="1.0em"></mspace></mtd></mtr></mtable><mo stretchy="true" form="postfix">]</mo></mrow></mrow><annotation encoding="application/x-tex">
\begin{bmatrix}
u' \\
v' \\
1 \\
\quad
\end{bmatrix}
=
\begin{bmatrix}
 h_{11} &   h_{12}  & h_{13} \\
 h_{21} &   h_{22}  & h_{23} \\
 h_{31} &   h_{32}  & h_{33}
\end{bmatrix}
\begin{bmatrix}
u \\
v \\
1 \\
\quad
\end{bmatrix}</annotation></semantics></math></p>

---

## Learning Homography

* Fun! But how can the DNN learn those parameters without labels?
* What if we used photometric loss (image similarity)?
* Given a pair of images, predict $H$, transform the first image, and compare to the second
  * The L1 pixel-wise error penalizes the DNN if the parameters it predicts do not align the images

---

## Read the Paper

<div class="col">
<img style="width: 70%" class="r-stretch" src="./figures/UnsupervisedDeepHomography_fig2.png" />
<br/>
<small> See Figure 2</small>
</div>

---

## Differentiability

* The authors put in some work to make the transformation differentiable
  * That's the Tensor DLT layer
  * Tensor Direct Linear Transform
* So this solution did require some effort

---

## Where is the Data?

* The idea sounds good, but we still need matching image pairs
  * But we can just make them from any image, we don't need labels
* For the authors, they were using satellite imagery

---

## Alignment

* If we can train a network to align points, then we can match edges of images
* So how about stitching together a larger images from several smaller ones?
  * For example, if you spin around taking images with your phone, can be put them together into a panorama?

---

## Image Stitching

* [Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images](https://arxiv.org/abs/2106.12859) did just that, from 2021
* If homographies can be learned between images, it is intuitive that this should work
* However, there is no dataset for this
  * So the authors used unsupervised learning

---

## Overview

<div class="col">
<img style="width: 70%" class="r-stretch" src="./figures/UnsupervisedDeepStitching_fig1.png" />
<br/>
<small>See Figure 1</small>
</div>

---

## Multiple Networks

* The deep image stitching work uses multiple DNNs
  * First, to align different images
  * Next, to deform images to match, and then to reconstruct images and "enhance the resolution"
    * Nowadays, you have probably seen plenty of super-resolution networks
    * Those are another example of unsupervised DNNs!

---

## Low Res, High Res

* Receptive fields cover more content if the resolution is small
  * So the deformation and stitching rulres are done on low resolution images
  * Those features are then given to an up-sampling network, which produces the output

---

## Dataset

* Of course it is possible to simply cut and warp existing data
  * But that doesn't prove things work in the real world
* So the authors simply grabbed some videos
  * Again, if you don't need labels, then any data can work
* They only needed to ensure that the frames had some overlap
  * By varying their sampling rate, they can control the overlap, and thus the difficulty

---

## Take aways

* Look at the images in the paper and you'll see that this the kind of thing on your smart phones today
* Why is this interesting?
  * Unsupervised learning can be applied in many ways
  * Both for pretraining powerful feature extracting, and to implement complicated functions
* This gets around deep learnings biggest weakness, which is the need for massive amounts of data

<!--
Other kinds of self-supervised learning.
########### Motion and real world structure
2017
Unsupervised Learning of Depth and Ego-Motion from Video" by Zhou et al.
https://openaccess.thecvf.com/content_cvpr_2017/html/Zhou_Unsupervised_Learning_of_CVPR_2017_paper.html

2017
Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model
Nguyen et al.
https://arxiv.org/abs/1709.03966

Unsupervised Deep Image Stitching: Reconstructing Stitched Features to Images
https://arxiv.org/abs/2106.12859

-->