CS 461 Homework 5

This homework won’t ask you to implement anything serious, just run some tests and make some graphs. The goal is to get you to think about feature extraction and feature spaces by interacting with them. For lecture 24, there were a large number of trained models. Let’s take the 50 epoch resnet model and the rbf kernel SVM trained with its feature vector and do some experiments.

The trained model is here.

The trained SVM pickle is here.

You can reload them as in the code used in lecture 24. (but example code will be given for this homework, so continue reading)

You will also need some example digits. The first twenty test images from digits are here. Their classes are (7 2 1 0 4 1 4 9 5 9 0 6 9 0 1 5 9 7 3 4).

The first twenty training images are here, with classes (5 0 4 1 9 2 1 3 1 4 3 5 3 6 1 7 2 8 6 9).

The example code loads any two digits and slowly morphs the first into the second, by sampling in decreasing part from the first and increasing from the second.

    print("Loading data")
    first = np.array(Image.open(args.first))/255.0
    second = np.array(Image.open(args.second))/255.0
    # Make a blending of the images in 101 steps, from fully first to fully second
    blends = []
    for i in range(101):
        alpha = (100 - i) / 100
        blend = first * alpha + second * (1 - alpha)
        blends.append(blend)
    data = np.array(blends)

Deliverables

This homework asks for answers to two open-ended questions and for several figures. The first question should be done before loading any numbers.

1. (20 points) Before loading any numbers (or reading ahead), how do you think the SVM and residual network will classify the blended digits? Will they guess wildly when presented an image that is an equal blend of two different digits? Explain, in a few sentences.

2. (20 points) Load a few pairs of digits to see if your intuition is correct or incorrect. If your supposition was correct, how could you verify your thought process? If your supposition was incorrect, where do you think you erred?

3. (20 points) Choose two different digits in either the training or the testing set and get their blended results. Plot the scores from the SVM and the DNN for the two correct classes (with score on the y axis and blend step on the x axis). You will need to modify the code to see all of the correct values. SVM scores are given by svm.decision_function and the DNN scores come from the model directly, in this code:

    # SVM Classification
    svm_classes = svm.predict(features)
    svm_y_hat = svm.decision_function(features)
    print(f"SVM scores {svm_y_hat}")
    print(f"SVM classes {svm_classes}")

    # DNN classification
    y_hat = model(data)
    classes = torch.argmax(y_hat, dim=1)

4. (20 points) Search for two digits where either the DNN or the SVM (or both) predict a third, nonpresent character. Provide the two digits (testing or training set and their index) and save an image of the blended figure where the SVM and/or DNN hallucinated a new digit. Also plot the scores vs blend step of the classifiers for the two correct classes and the incorrect third class.

5. (20 points) Are there any pairs of digits belonging to the same class where either the DNN or SVM (or both) predict a different character while they are being blended? You can either search manually or write a script. If you find such a case, save the image where the SVM and/or DNN hallucinated a new digit.

6. (extra 20 points) Write your own dataset of 10 digits and test classification with the SVM and DNN. You will need to convert your images to white on black, and scale to 28x28 pixels. Are any misclassified?