{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2fWDjZRP5GZ"
      },
      "source": [
        "# Cake classification (part 1)\n",
        "\n",
        "Image recognition is a very important application of machine learning.\n",
        "There are two main approaches:\n",
        "1. with hand-crafted image features processed by a classification\n",
        "  model;\n",
        "2. with deep-learning, with a neural network that directly\n",
        "  processes the image pixels.  Convolutional neural networks (CNN) are\n",
        "  the most suitable architecture for this application.\n",
        "\n",
        "## Lab activity\n",
        "\n",
        "The lab is divided into two parts:\n",
        "* first, low level features are extracted and simple MLPs are built to try and classify the images\n",
        "* secondly, a deep learning approach with transfer learning from ResNet will be used\n",
        "\n",
        "The data set contains 120 images for each of 15 kinds of cake.  For\n",
        "each class 100 images are in the training set and 20 form the test\n",
        "set.  All the images have been resized to $224 \\times 224$ pixels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code: part 1\n",
        "\n",
        "Imports:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "metadata": {
        "id": "98dnoISnRooW"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Dataset\n",
        "The dataset loaders are prepared and a sample batch is visualized"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "metadata": {
        "id": "Ws62ZJ0pRgip"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['tiramisu', 'macarons', 'donuts', 'red_velvet_cake', 'cannoli', 'ice_cream', 'red_velvet_cake', 'cannoli', 'panna_cotta', 'panna_cotta', 'chocolate_mousse', 'apple_pie', 'macarons', 'waffles', 'cannoli', 'carrot_cake', 'tiramisu', 'waffles', 'red_velvet_cake', 'donuts', 'apple_pie', 'macarons', 'apple_pie', 'cup_cakes', 'donuts', 'creme_brulee', 'carrot_cake', 'waffles', 'ice_cream', 'waffles', 'churros', 'cup_cakes']\n",
            "torch.Size([32, 3, 224, 224])\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 1364140 characters]",
            "text/plain": [
              "<Figure size 1280x960 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "transform = torchvision.transforms.Compose([\n",
        "    torchvision.transforms.RandomHorizontalFlip(),\n",
        "    torchvision.transforms.ToTensor()\n",
        "])\n",
        "\n",
        "train_set = torchvision.datasets.ImageFolder(\"./cake-classification/images/train\", transform=transform)\n",
        "train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True, drop_last=True, num_workers=2)\n",
        "\n",
        "test_set = torchvision.datasets.ImageFolder(\"./cake-classification/images/test\", transform=torchvision.transforms.ToTensor())\n",
        "test_loader = torch.utils.data.DataLoader(test_set, batch_size=32, shuffle=False, drop_last=False, num_workers=2)\n",
        "\n",
        "\n",
        "images, labels = next(iter(train_loader))\n",
        "plt.figure(dpi=200)\n",
        "grid = torchvision.utils.make_grid(images, nrow=8, normalize=True).permute(1, 2, 0)\n",
        "plt.axis(\"off\")\n",
        "plt.imshow(grid)\n",
        "print([train_set.classes[i] for i in labels])\n",
        "print(images.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjMTPP9X_d56"
      },
      "source": [
        "### Feature extraction\n",
        "\n",
        "For feature extraction we will use low-level features:\n",
        "1. color histograms;\n",
        "2. edge direction histograms;\n",
        "3. co-occurence matrices.\n",
        "\n",
        "Different concatenations of these will be tested against different models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "metadata": {
        "id": "uy39vGgvP3qg"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([7, 729])\n"
          ]
        }
      ],
      "source": [
        "def color_histogram(img, bins=64):\n",
        "    n = img.shape[0]\n",
        "    img = (img * bins).long().clamp(0, bins - 1)\n",
        "    img = img + torch.arange(0, 3 * n * bins, bins, device=img.device).view(n, 3, 1, 1)\n",
        "    hist = torch.bincount(img.view(-1), minlength=(3 * n * bins)).view(n, 3 * bins)\n",
        "    return hist / img[0, 1].numel()\n",
        "\n",
        "\n",
        "def edge_direction_histogram(img, bins=64):\n",
        "    n = img.shape[0]\n",
        "    SOBEL = torch.tensor([\n",
        "        [[[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]],\n",
        "        [[[-1, -2, -1], [0, 0, 0], [1, 2, 1]]]\n",
        "    ], dtype=img.dtype, device=img.device)\n",
        "    gray = img.mean(1, True)\n",
        "    grad = torch.nn.functional.conv2d(gray, SOBEL)\n",
        "    magnitude = grad.norm(dim=1)\n",
        "    angle = torch.atan2(grad[:, 0], grad[:, 1])\n",
        "    directions = (bins * angle / (2 * torch.pi)).long() % bins\n",
        "    directions = directions + torch.arange(0, n * bins, bins, device=img.device).view(n, 1, 1)\n",
        "    hist = torch.bincount(directions.view(-1), weights=magnitude.view(-1), minlength=n * bins).view(n, bins)\n",
        "    hist = hist / hist.sum(1, True).clamp(1e-6)\n",
        "    return hist\n",
        "\n",
        "\n",
        "def _cooccurrence_matrix_dir(values, bins, di, dj):\n",
        "    b, m, n = values.shape\n",
        "    codes = values[:, :m - di, :n - dj] + bins * values[:, di:, dj:]\n",
        "    codes += torch.arange(0, b * (bins ** 2), bins ** 2, device=values.device).view(b, 1, 1)\n",
        "    entries = torch.bincount(codes.view(-1), minlength=b * (bins ** 2)).view(b, bins, bins)\n",
        "    return entries\n",
        "\n",
        "\n",
        "def rgb_cooccurrence_matrix(img, quantization=3, distance=10):\n",
        "    n = img.shape[0]\n",
        "    img = (img * quantization).long().clamp(0, quantization - 1)\n",
        "    img = (img * (torch.arange(3, device=img.device) ** quantization).view(1, 3, 1, 1)).sum(1)\n",
        "    bins = quantization ** 3            # the num of features is actually the square of this cause i do 2 mats\n",
        "    mat = _cooccurrence_matrix_dir(img, bins, distance, 0)\n",
        "    mat += _cooccurrence_matrix_dir(img, bins, 0, distance)\n",
        "    mat = (mat + mat.permute(0, 2, 1)).view(n, -1)\n",
        "    mat = mat / mat.sum(1, True)\n",
        "    return mat\n",
        "\n",
        "\n",
        "# Try to see how the co-occurrence matrix works\n",
        "x = torch.rand(7, 3, 224, 224)\n",
        "h = rgb_cooccurrence_matrix(x)\n",
        "print(h.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfGCbAHLAVd3"
      },
      "source": [
        "### Classifier 1: logistic regression\n",
        "\n",
        "The first try is a simple logistic regression with color histogram as features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {
        "id": "6SqpcCHSitCA"
      },
      "outputs": [],
      "source": [
        "LR = 0.01\n",
        "DEVICE = (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "lin_reg_model = torch.nn.Linear(192, len(train_set.classes))            # the histo has 64 bins, so for 3 channels you have 192 features\n",
        "lin_reg_model.to(DEVICE)\n",
        "\n",
        "opti_lin_reg = torch.optim.Adam(lin_reg_model.parameters(), lr=LR)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()              # you avoid the softmax if you use the cross entropy! You can use it later to compute probabilities"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m4Ib7zQYAhdu"
      },
      "source": [
        "Usual training loop with feature extraction. A train function is defined since it is model independent and can be reused with different models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {
        "id": "pfDzNABikhIZ"
      },
      "outputs": [],
      "source": [
        "def train(model, optimizer, loss_fun, train_loader, epochs=20, features_to_use=[\"color_histo\"], verbose=True, plotting=False):\n",
        "    steps = []\n",
        "    accuracies = []\n",
        "    losses = []\n",
        "\n",
        "    step = 0\n",
        "    for epoch in range(epochs):\n",
        "        for imgs, labels in train_loader:\n",
        "            imgs = imgs.to(DEVICE)\n",
        "            labels = labels.to(DEVICE)\n",
        "            optimizer.zero_grad()\n",
        "            features = []\n",
        "            if \"color_histo\" in features_to_use:\n",
        "                features.append(color_histogram(imgs))\n",
        "            if \"edge_dir_histo\" in features_to_use:\n",
        "                features.append(edge_direction_histogram(imgs))\n",
        "            if \"rgb_cooc_mat\" in features_to_use:\n",
        "                features.append(rgb_cooccurrence_matrix(imgs))\n",
        "            features = torch.cat(features, dim=1)  # concatenate the features along the feature dimension\n",
        "            logits = model(features)\n",
        "            loss = loss_fun(logits, labels)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            step += 1\n",
        "            if step % 100 == 0:\n",
        "                predictions = logits.argmax(dim=1)\n",
        "                accuracy = (predictions == labels).float().mean()\n",
        "                steps.append(step)\n",
        "                accuracies.append(accuracy.item())\n",
        "                losses.append(loss.item())\n",
        "                if verbose:\n",
        "                    print(f\"{step} [{epoch}] Loss: {loss.item():.4f} Acc: {accuracy.item() * 100:.1f}\")\n",
        "    \n",
        "    if plotting:\n",
        "        plt.figure(figsize=(8, 5))\n",
        "        fig, ax1 = plt.subplots(figsize=(8, 5))\n",
        "        set1 = plt.get_cmap('Set1')\n",
        "        color = set1(2)\n",
        "        ax1.set_xlabel(\"Iterations\")\n",
        "        ax1.set_ylabel(\"Accuracy\", color=color,  fontsize=14)\n",
        "        ax1.plot(steps, accuracies, label=\"Accuracy\", color=color, linewidth=2)\n",
        "        ax1.tick_params(axis='y', labelcolor=color)\n",
        "        ax1.set_ylim(0, 1)  # accuracy between 0 and 1\n",
        "\n",
        "        ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis\n",
        "\n",
        "        color = set1(1)\n",
        "        ax2.set_ylabel(\"Loss\", color=color,  fontsize=14)\n",
        "        ax2.plot(steps, losses, label=\"Loss\", color=color, linewidth=2)\n",
        "        ax2.tick_params(axis='y', labelcolor=color)\n",
        "\n",
        "        fig.tight_layout()\n",
        "        plt.title(\"Accuracy and Loss over Iterations\")\n",
        "        plt.show()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.6101 Acc: 21.9\n",
            "200 [4] Loss: 2.5283 Acc: 21.9\n",
            "300 [6] Loss: 2.6324 Acc: 12.5\n",
            "400 [8] Loss: 2.6082 Acc: 15.6\n",
            "500 [10] Loss: 2.5289 Acc: 25.0\n",
            "600 [13] Loss: 2.3678 Acc: 31.2\n",
            "700 [15] Loss: 2.4297 Acc: 18.8\n",
            "800 [17] Loss: 2.4140 Acc: 25.0\n",
            "900 [19] Loss: 2.2839 Acc: 31.2\n",
            "1000 [21] Loss: 2.4433 Acc: 28.1\n",
            "1100 [23] Loss: 2.1881 Acc: 43.8\n",
            "1200 [26] Loss: 2.1698 Acc: 28.1\n",
            "1300 [28] Loss: 2.1811 Acc: 25.0\n",
            "1400 [30] Loss: 2.1847 Acc: 40.6\n",
            "1500 [32] Loss: 2.2421 Acc: 37.5\n",
            "1600 [34] Loss: 2.2887 Acc: 25.0\n",
            "1700 [36] Loss: 2.2037 Acc: 34.4\n",
            "1800 [39] Loss: 2.1898 Acc: 21.9\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 68556 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "train(lin_reg_model, opti_lin_reg, loss_fun, train_loader, epochs=40, features_to_use=['color_histo'], verbose=True, plotting=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRuYA_CIA_Iu"
      },
      "source": [
        "The same approach is used to evaluate the model. Appropriate functions are defined for later use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {
        "id": "QTx8s1pdnNYE"
      },
      "outputs": [],
      "source": [
        "def test_conf_mat(model, test_loader, nclasses=15, features_to_use=[\"color_histo\"], verbose=True):\n",
        "    conf_mat = torch.zeros(nclasses, nclasses, dtype=torch.long)\n",
        "\n",
        "    for imgs, labels in test_loader:\n",
        "        imgs = imgs.to(DEVICE)\n",
        "        labels = labels.to(DEVICE)\n",
        "\n",
        "        features = []\n",
        "        if \"color_histo\" in features_to_use:\n",
        "            features.append(color_histogram(imgs))\n",
        "        if \"edge_dir_histo\" in features_to_use:\n",
        "            features.append(edge_direction_histogram(imgs))\n",
        "        if \"rgb_cooc_mat\" in features_to_use:\n",
        "            features.append(rgb_cooccurrence_matrix(imgs))\n",
        "        features = torch.cat(features, dim=1)\n",
        "        \n",
        "        with torch.no_grad():\n",
        "            logits = model(features)\n",
        "        predictions = torch.argmax(logits, dim=1)\n",
        "\n",
        "        codes = labels * nclasses + predictions\n",
        "        hist = torch.bincount(codes.cpu(), minlength=nclasses ** 2)\n",
        "        conf_mat += hist.view(nclasses, nclasses)\n",
        "\n",
        "    accuracy = conf_mat.diag().sum() / conf_mat.sum()\n",
        "    if verbose:\n",
        "        print(f\"Accuracy: {100 * accuracy:.1f}%\")\n",
        "\n",
        "    # since the confusion matrix is evaluated for the accuracy, it can be safely returned\n",
        "    return conf_mat\n",
        "\n",
        "def plot_conf_mat(conf_mat, classes):\n",
        "    conf_mat_norm = conf_mat / conf_mat.sum(1)\n",
        "    plt.figure(dpi=100)\n",
        "    plt.imshow(conf_mat_norm, cmap=\"Blues\")\n",
        "    plt.xticks(range(len(classes)), classes, rotation=90)\n",
        "    plt.yticks(range(len(classes)), classes)\n",
        "    plt.colorbar()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 23.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 68432 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "conf_mat = test_conf_mat(lin_reg_model, test_loader, nclasses=len(train_set.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat, train_set.classes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.5928 Acc: 21.9\n",
            "200 [4] Loss: 2.4874 Acc: 34.4\n",
            "300 [6] Loss: 2.3339 Acc: 21.9\n",
            "400 [8] Loss: 2.4752 Acc: 6.2\n",
            "500 [10] Loss: 2.3864 Acc: 18.8\n",
            "600 [13] Loss: 2.3554 Acc: 25.0\n",
            "700 [15] Loss: 2.2299 Acc: 37.5\n",
            "800 [17] Loss: 2.2286 Acc: 31.2\n",
            "900 [19] Loss: 2.5024 Acc: 21.9\n",
            "1000 [21] Loss: 2.0614 Acc: 56.2\n",
            "1100 [23] Loss: 2.1313 Acc: 31.2\n",
            "1200 [26] Loss: 2.2754 Acc: 25.0\n",
            "1300 [28] Loss: 2.0484 Acc: 37.5\n",
            "1400 [30] Loss: 2.3821 Acc: 21.9\n",
            "1500 [32] Loss: 2.1431 Acc: 34.4\n",
            "1600 [34] Loss: 2.1208 Acc: 43.8\n",
            "1700 [36] Loss: 2.0692 Acc: 34.4\n",
            "1800 [39] Loss: 2.1637 Acc: 37.5\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 78120 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 25.7%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 68304 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# edge histo and color histo together\n",
        "lin_reg_double_histo_model = torch.nn.Linear(256, len(train_set.classes))            # 192 from color histo + 64 from edge dir histo\n",
        "lin_reg_double_histo_model.to(DEVICE)\n",
        "\n",
        "opti_lin_reg_dobule_histo = torch.optim.Adam(lin_reg_double_histo_model.parameters(), lr=LR)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "train(lin_reg_double_histo_model, opti_lin_reg_dobule_histo, loss_fun, train_loader, epochs=40, features_to_use=['color_histo', 'edge_dir_histo'], verbose=True, plotting=True)\n",
        "conf_mat = test_conf_mat(lin_reg_double_histo_model, test_loader, nclasses=len(train_set.classes), features_to_use=['color_histo', 'edge_dir_histo'], verbose=True)\n",
        "plot_conf_mat(conf_mat, train_set.classes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.5483 Acc: 25.0\n",
            "200 [4] Loss: 2.5435 Acc: 21.9\n",
            "300 [6] Loss: 2.3617 Acc: 28.1\n",
            "400 [8] Loss: 2.3053 Acc: 28.1\n",
            "500 [10] Loss: 2.2162 Acc: 37.5\n",
            "600 [13] Loss: 2.1886 Acc: 31.2\n",
            "700 [15] Loss: 2.1865 Acc: 37.5\n",
            "800 [17] Loss: 2.3332 Acc: 21.9\n",
            "900 [19] Loss: 2.0357 Acc: 43.8\n",
            "1000 [21] Loss: 2.2015 Acc: 34.4\n",
            "1100 [23] Loss: 2.0358 Acc: 40.6\n",
            "1200 [26] Loss: 2.0926 Acc: 37.5\n",
            "1300 [28] Loss: 2.0287 Acc: 34.4\n",
            "1400 [30] Loss: 1.9174 Acc: 40.6\n",
            "1500 [32] Loss: 1.8590 Acc: 46.9\n",
            "1600 [34] Loss: 2.1841 Acc: 34.4\n",
            "1700 [36] Loss: 1.9269 Acc: 40.6\n",
            "1800 [39] Loss: 1.8771 Acc: 43.8\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 70944 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 32.7%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 67532 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# color histo and co-occurrence matrix together\n",
        "lin_reg_color_cooc_model = torch.nn.Linear(921, len(train_set.classes))            # 192 from color histo + 729 from co-occurrence matrix (3 channels, 3^3 = 27 bins, 27^2 = 729)\n",
        "lin_reg_color_cooc_model.to(DEVICE)\n",
        "\n",
        "opti_lin_reg_color_cooc = torch.optim.Adam(lin_reg_color_cooc_model.parameters(), lr=LR)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "train(lin_reg_color_cooc_model, opti_lin_reg_color_cooc, loss_fun, train_loader, epochs=40, features_to_use=['color_histo', 'rgb_cooc_mat'], verbose=True, plotting=True)\n",
        "conf_mat = test_conf_mat(lin_reg_color_cooc_model, test_loader, nclasses=len(train_set.classes), features_to_use=['color_histo', 'rgb_cooc_mat'], verbose=True)\n",
        "plot_conf_mat(conf_mat, train_set.classes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.5561 Acc: 18.8\n",
            "200 [4] Loss: 2.5637 Acc: 18.8\n",
            "300 [6] Loss: 2.2912 Acc: 28.1\n",
            "400 [8] Loss: 2.3009 Acc: 18.8\n",
            "500 [10] Loss: 2.2874 Acc: 25.0\n",
            "600 [13] Loss: 1.9833 Acc: 46.9\n",
            "700 [15] Loss: 2.2115 Acc: 28.1\n",
            "800 [17] Loss: 1.8444 Acc: 53.1\n",
            "900 [19] Loss: 2.3520 Acc: 25.0\n",
            "1000 [21] Loss: 2.0090 Acc: 37.5\n",
            "1100 [23] Loss: 2.1647 Acc: 37.5\n",
            "1200 [26] Loss: 2.0971 Acc: 37.5\n",
            "1300 [28] Loss: 1.8186 Acc: 56.2\n",
            "1400 [30] Loss: 1.9743 Acc: 37.5\n",
            "1500 [32] Loss: 2.0151 Acc: 40.6\n",
            "1600 [34] Loss: 1.8253 Acc: 56.2\n",
            "1700 [36] Loss: 1.9371 Acc: 53.1\n",
            "1800 [39] Loss: 1.9791 Acc: 37.5\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 76856 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 32.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 67468 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# all low level features together\n",
        "lin_reg_all_model = torch.nn.Linear(985, len(train_set.classes))            # 192 from color histo + 64 from edge histo + 729 from co-occurrence matrix\n",
        "lin_reg_all_model.to(DEVICE)\n",
        "\n",
        "opti_lin_reg_all = torch.optim.Adam(lin_reg_all_model.parameters(), lr=LR)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "train(lin_reg_all_model, opti_lin_reg_all, loss_fun, train_loader, epochs=40, features_to_use=['color_histo', 'edge_dir_histo', 'rgb_cooc_mat'], verbose=True, plotting=True)\n",
        "conf_mat = test_conf_mat(lin_reg_all_model, test_loader, nclasses=len(train_set.classes), features_to_use=['color_histo', 'edge_dir_histo', 'rgb_cooc_mat'], verbose=True)\n",
        "plot_conf_mat(conf_mat, train_set.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Classifier 2: mlp with 3 hidden layers\n",
        "\n",
        "Since the best result is achieved using all features, now a simple MLP with 3 hidden layers (using all features) is implemented."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 99,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.5158 Acc: 9.4\n",
            "200 [4] Loss: 2.3574 Acc: 12.5\n",
            "300 [6] Loss: 2.2352 Acc: 9.4\n",
            "400 [8] Loss: 2.2580 Acc: 28.1\n",
            "500 [10] Loss: 2.3138 Acc: 40.6\n",
            "600 [13] Loss: 2.1258 Acc: 37.5\n",
            "700 [15] Loss: 2.0141 Acc: 31.2\n",
            "800 [17] Loss: 2.0852 Acc: 28.1\n",
            "900 [19] Loss: 2.1137 Acc: 28.1\n",
            "1000 [21] Loss: 2.1102 Acc: 25.0\n",
            "1100 [23] Loss: 1.9853 Acc: 37.5\n",
            "1200 [26] Loss: 1.6097 Acc: 43.8\n",
            "1300 [28] Loss: 1.7239 Acc: 34.4\n",
            "1400 [30] Loss: 1.7214 Acc: 46.9\n",
            "1500 [32] Loss: 1.7210 Acc: 43.8\n",
            "1600 [34] Loss: 1.4413 Acc: 43.8\n",
            "1700 [36] Loss: 1.8158 Acc: 43.8\n",
            "1800 [39] Loss: 1.6361 Acc: 40.6\n",
            "1900 [41] Loss: 1.0003 Acc: 62.5\n",
            "2000 [43] Loss: 1.2873 Acc: 62.5\n",
            "2100 [45] Loss: 1.6261 Acc: 46.9\n",
            "2200 [47] Loss: 1.3631 Acc: 56.2\n",
            "2300 [49] Loss: 0.9187 Acc: 71.9\n",
            "2400 [52] Loss: 1.1633 Acc: 56.2\n",
            "2500 [54] Loss: 0.9676 Acc: 78.1\n",
            "2600 [56] Loss: 0.6340 Acc: 81.2\n",
            "2700 [58] Loss: 0.9795 Acc: 68.8\n",
            "2800 [60] Loss: 1.3316 Acc: 59.4\n",
            "2900 [63] Loss: 0.7851 Acc: 78.1\n",
            "3000 [65] Loss: 0.5967 Acc: 87.5\n",
            "3100 [67] Loss: 0.6481 Acc: 75.0\n",
            "3200 [69] Loss: 0.7084 Acc: 71.9\n",
            "3300 [71] Loss: 0.8941 Acc: 71.9\n",
            "3400 [73] Loss: 0.6081 Acc: 90.6\n",
            "3500 [76] Loss: 0.7183 Acc: 71.9\n",
            "3600 [78] Loss: 0.8093 Acc: 75.0\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 83272 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 31.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 68252 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "mlp = torch.nn.Sequential(\n",
        "    torch.nn.Linear(985, 512),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(512, 256),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(256, 128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128, len(train_set.classes))\n",
        ")\n",
        "mlp.to(DEVICE)\n",
        "\n",
        "opti_mlp = torch.optim.Adam(mlp.parameters(), lr=0.001)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "train(mlp, opti_mlp, loss_fun, train_loader, epochs=80, features_to_use=['color_histo', 'edge_dir_histo', 'rgb_cooc_mat'], verbose=True, plotting=True)\n",
        "conf_mat = test_conf_mat(mlp, test_loader, nclasses=len(train_set.classes), features_to_use=['color_histo', 'edge_dir_histo', 'rgb_cooc_mat'], verbose=True)\n",
        "plot_conf_mat(conf_mat, train_set.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code: part 2\n",
        "\n",
        "In this part, a transfer-learning approach from a pre-trained ResNet is implemented."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 174,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "ResNet(\n",
            "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
            "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  (relu): ReLU(inplace=True)\n",
            "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
            "  (layer1): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer2): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer3): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (layer4): Sequential(\n",
            "    (0): BasicBlock(\n",
            "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (downsample): Sequential(\n",
            "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
            "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      )\n",
            "    )\n",
            "    (1): BasicBlock(\n",
            "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "      (relu): ReLU(inplace=True)\n",
            "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
            "  (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
            ")\n",
            "11,689,512 parameters\n"
          ]
        }
      ],
      "source": [
        "weights = torchvision.models.ResNet18_Weights.DEFAULT\n",
        "res_net = torchvision.models.resnet18(weights=weights) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "print(res_net)\n",
        "ps = sum(param.nelement() for param in res_net.parameters())\n",
        "print(f\"{ps:,d} parameters\")\n",
        "\n",
        "preprocess = weights.transforms()           # needed for appropriate image preprocessing\n",
        "\n",
        "imagenet_classes = torchvision.models.resnet._IMAGENET_CATEGORIES"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Dataset\n",
        "\n",
        "Data loaders are configured with the necessary preprocessing to work with the pre-trained ResNet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 175,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['apple_pie', 'churros', 'carrot_cake', 'cup_cakes', 'tiramisu', 'creme_brulee', 'churros', 'carrot_cake', 'carrot_cake', 'cup_cakes', 'cannoli', 'cup_cakes', 'waffles', 'tiramisu', 'tiramisu', 'macarons', 'red_velvet_cake', 'cup_cakes', 'red_velvet_cake', 'carrot_cake', 'chocolate_cake', 'ice_cream', 'macarons', 'chocolate_cake', 'churros', 'cup_cakes', 'ice_cream', 'cannoli', 'creme_brulee', 'cup_cakes', 'waffles', 'chocolate_mousse']\n",
            "torch.Size([32, 3, 224, 224])\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 1294628 characters]",
            "text/plain": [
              "<Figure size 1280x960 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "transform_deep_net = torchvision.transforms.Compose([\n",
        "    torchvision.transforms.RandomHorizontalFlip(),\n",
        "    preprocess\n",
        "])\n",
        "\n",
        "train_set_net = torchvision.datasets.ImageFolder(\"./cake-classification/images/train\", transform=transform_deep_net)\n",
        "train_loader_net = torch.utils.data.DataLoader(train_set_net, batch_size=32, shuffle=True, drop_last=True, num_workers=2)\n",
        "\n",
        "test_set_net = torchvision.datasets.ImageFolder(\"./cake-classification/images/test\", transform=preprocess)\n",
        "test_loader_net = torch.utils.data.DataLoader(test_set_net, batch_size=32, shuffle=False, drop_last=False, num_workers=2)\n",
        "\n",
        "# test to see if the data is loaded correctly\n",
        "images, labels = next(iter(train_loader_net))\n",
        "plt.figure(dpi=200)\n",
        "grid = torchvision.utils.make_grid(images, nrow=8, normalize=True).permute(1, 2, 0)\n",
        "plt.axis(\"off\")\n",
        "plt.imshow(grid)\n",
        "print([train_set.classes[i] for i in labels])\n",
        "print(images.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Model definition and training\n",
        "In the following cells the model is defined and trained in a similar way of what has been done before"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 177,
      "metadata": {},
      "outputs": [],
      "source": [
        "loss_fun = torch.nn.CrossEntropyLoss()\n",
        "loss_fun.to(DEVICE)\n",
        "\n",
        "# 512 are the res_net.fc.in_features\n",
        "res_net.fc = torch.nn.Linear(512, len(train_set_net.classes))\n",
        "res_net = res_net.to(DEVICE)\n",
        "\n",
        "# here I'm not passing all res_net.parameters(), but ONLY the new layer that i added (the last one, the fully connected layer)\n",
        "# That's because I want to use the pretrained weights of the resnet18, and only train the last layer\n",
        "opti_res_net = torch.optim.Adam(res_net.fc.parameters(), lr=0.001)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 178,
      "metadata": {},
      "outputs": [],
      "source": [
        "def train_net(net, optimizer, train_loader, epochs=10, verbose=True, plotting=False):\n",
        "    steps = []\n",
        "    accuracies = []\n",
        "    losses = []\n",
        "\n",
        "    # set the net to training mode\n",
        "    net.train()\n",
        "    step = 0\n",
        "    for epoch in range(epochs):\n",
        "        for images, labels in train_loader:\n",
        "            images = images.to(DEVICE)\n",
        "            labels = labels.to(DEVICE)\n",
        "            optimizer.zero_grad()\n",
        "            logits = net(images)\n",
        "            loss = loss_fun(logits, labels)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            step += 1\n",
        "            if step % 100 == 0:\n",
        "                predictions = logits.argmax(1)\n",
        "                accuracy = (predictions == labels).float().mean()\n",
        "                steps.append(step)\n",
        "                accuracies.append(accuracy.item())\n",
        "                losses.append(loss.item())\n",
        "                if verbose:\n",
        "                    print(f\"{step} [{epoch}]  Loss: {loss.item():.4f}  Acc: {accuracy.item() * 100:.1f}%\")\n",
        "    \n",
        "    if plotting:\n",
        "        plt.figure(figsize=(8, 5))\n",
        "        fig, ax1 = plt.subplots(figsize=(8, 5))\n",
        "        set1 = plt.get_cmap('Set1')\n",
        "        color = set1(2)\n",
        "        ax1.set_xlabel(\"Iterations\")\n",
        "        ax1.set_ylabel(\"Accuracy\", color=color,  fontsize=14)\n",
        "        ax1.plot(steps, accuracies, label=\"Accuracy\", color=color, linewidth=2)\n",
        "        ax1.tick_params(axis='y', labelcolor=color)\n",
        "        ax1.set_ylim(0, 1)  # accuracy between 0 and 1\n",
        "\n",
        "        ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis\n",
        "\n",
        "        color = set1(1)\n",
        "        ax2.set_ylabel(\"Loss\", color=color,  fontsize=14)\n",
        "        ax2.plot(steps, losses, label=\"Loss\", color=color, linewidth=2)\n",
        "        ax2.tick_params(axis='y', labelcolor=color)\n",
        "\n",
        "        fig.tight_layout()\n",
        "        plt.title(\"Accuracy and Loss over Iterations\")\n",
        "        plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 179,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 0.9630  Acc: 78.1%\n",
            "200 [4]  Loss: 0.3139  Acc: 96.9%\n",
            "300 [6]  Loss: 0.2416  Acc: 96.9%\n",
            "400 [8]  Loss: 0.3222  Acc: 96.9%\n",
            "500 [10]  Loss: 0.4250  Acc: 84.4%\n",
            "600 [13]  Loss: 0.1529  Acc: 100.0%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 57032 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "train_net(res_net, opti_res_net, train_loader=train_loader_net, epochs=15, verbose=True, plotting=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Model evalutaion\n",
        "The network obtained with transfer learning is evaluated on the test set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 180,
      "metadata": {},
      "outputs": [],
      "source": [
        "def test_conf_mat_res_net(net, test_loader, nclasses=15, verbose=True):\n",
        "    conf_mat = torch.zeros(nclasses, nclasses, dtype=torch.long)\n",
        "\n",
        "    net.eval()  # set the net to evaluation mode\n",
        "\n",
        "    for imgs, labels in test_loader:\n",
        "        imgs = imgs.to(DEVICE)\n",
        "        labels = labels.to(DEVICE)\n",
        "        with torch.no_grad():\n",
        "            logits = net(imgs)\n",
        "        predictions = torch.argmax(logits, dim=1)\n",
        "\n",
        "        codes = labels * nclasses + predictions\n",
        "        hist = torch.bincount(codes.cpu(), minlength=nclasses ** 2)\n",
        "        conf_mat += hist.view(nclasses, nclasses).cpu()\n",
        "\n",
        "    accuracy = conf_mat.diag().sum() / conf_mat.sum()\n",
        "    if verbose:\n",
        "        print(f\"Accuracy: {100 * accuracy:.1f}%\")\n",
        "\n",
        "    # since the confusion matrix is evaluated for the accuracy, it can be safely returned\n",
        "    return conf_mat"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 181,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 93.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66360 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "conf_mat_net = test_conf_mat_res_net(res_net, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 182,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 1003436 characters]",
            "text/plain": [
              "<Figure size 1200x700 with 6 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "errors = []\n",
        "\n",
        "for images, labels in test_loader:\n",
        "    images = images.to(DEVICE)\n",
        "    labels = labels.to(DEVICE)\n",
        "    with torch.no_grad():\n",
        "        logits = res_net(images)\n",
        "    scores = logits.max(1).values\n",
        "    predictions = logits.argmax(1)\n",
        "    for l, p, s, im in zip(labels, predictions, scores, images):\n",
        "        if l != p:\n",
        "            errors.append((s, im.cpu(), l, p))\n",
        "\n",
        "errors.sort(reverse=True)\n",
        "fig, axes = plt.subplots(2, 3, figsize=(12, 7))\n",
        "\n",
        "for idx, (s, im, l, p) in enumerate(errors[:6]):\n",
        "    ax = axes[idx // 3, idx % 3]\n",
        "    img = im.permute(1, 2, 0) * 0.2 + 0.4\n",
        "    ax.imshow(img.clamp(0, 1))         # not 100% correct but roughly reverse the normalization so that i can see the picture\n",
        "    ax.set_title(train_set.classes[l] + \" -> \" + train_set.classes[p])\n",
        "    ax.axis(\"off\")\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Tranfer learning from precedent layers\n",
        "\n",
        "To implement transfer learning from precedent layers, some layers can be directly set to identities.\n",
        "\n",
        "First, the cut is applied before layer 4 (thus replacing layer 4 with identity)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 183,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 1.7474  Acc: 34.4%\n",
            "200 [4]  Loss: 1.5075  Acc: 62.5%\n",
            "300 [6]  Loss: 1.1059  Acc: 84.4%\n",
            "400 [8]  Loss: 1.2497  Acc: 56.2%\n",
            "500 [10]  Loss: 0.9484  Acc: 71.9%\n",
            "600 [13]  Loss: 0.7686  Acc: 84.4%\n",
            "700 [15]  Loss: 0.8682  Acc: 78.1%\n",
            "800 [17]  Loss: 0.5886  Acc: 90.6%\n",
            "900 [19]  Loss: 0.8435  Acc: 81.2%\n",
            "1000 [21]  Loss: 0.5548  Acc: 90.6%\n",
            "1100 [23]  Loss: 0.4487  Acc: 93.8%\n",
            "1200 [26]  Loss: 0.6881  Acc: 78.1%\n",
            "1300 [28]  Loss: 0.5622  Acc: 93.8%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 72808 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 86.0%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66716 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "res_net_early = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.DEFAULT) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "\n",
        "# replacing layer 4 with Identity\n",
        "res_net_early.layer4 = torch.nn.Identity()\n",
        "\n",
        "# 256 are the in_features remaining after layer 3\n",
        "res_net_early.fc = torch.nn.Linear(256, len(train_set_net.classes))\n",
        "res_net_early = res_net_early.to(DEVICE)\n",
        "\n",
        "opti_res_net_early = torch.optim.Adam(res_net_early.fc.parameters(), lr=0.01)\n",
        "\n",
        "train_net(res_net_early, opti_res_net_early, train_loader=train_loader_net, epochs=30, verbose=True, plotting=True)\n",
        "conf_mat_net = test_conf_mat_res_net(res_net_early, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The results are worse than before. Lets try not only replacing the last fully connected but add a couple."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 185,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 1.6283  Acc: 46.9%\n",
            "200 [4]  Loss: 1.1779  Acc: 56.2%\n",
            "300 [6]  Loss: 1.3363  Acc: 56.2%\n",
            "400 [8]  Loss: 0.4277  Acc: 84.4%\n",
            "500 [10]  Loss: 0.7261  Acc: 71.9%\n",
            "600 [13]  Loss: 0.5889  Acc: 71.9%\n",
            "700 [15]  Loss: 0.3656  Acc: 84.4%\n",
            "800 [17]  Loss: 0.5070  Acc: 87.5%\n",
            "900 [19]  Loss: 0.4119  Acc: 87.5%\n",
            "1000 [21]  Loss: 0.6729  Acc: 81.2%\n",
            "1100 [23]  Loss: 0.4051  Acc: 81.2%\n",
            "1200 [26]  Loss: 0.2813  Acc: 90.6%\n",
            "1300 [28]  Loss: 0.3159  Acc: 90.6%\n",
            "1400 [30]  Loss: 0.4121  Acc: 87.5%\n",
            "1500 [32]  Loss: 0.2990  Acc: 90.6%\n",
            "1600 [34]  Loss: 0.4212  Acc: 87.5%\n",
            "1700 [36]  Loss: 0.2435  Acc: 96.9%\n",
            "1800 [39]  Loss: 0.2824  Acc: 87.5%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 66072 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 82.0%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66604 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "res_net_early = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.DEFAULT) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "\n",
        "# replacing layer 4 with Identity\n",
        "res_net_early.layer4 = torch.nn.Identity()\n",
        "\n",
        "# Replace fc with a small MLP (256 -> 128 -> 64 -> num_classes)\n",
        "res_net_early.fc = torch.nn.Sequential(\n",
        "    torch.nn.Linear(256, 128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(train_set_net.classes))\n",
        ")\n",
        "res_net_early = res_net_early.to(DEVICE)\n",
        "\n",
        "opti_res_net_early = torch.optim.Adam(res_net_early.fc.parameters(), lr=0.01)\n",
        "\n",
        "train_net(res_net_early, opti_res_net_early, train_loader=train_loader_net, epochs=40, verbose=True, plotting=True)\n",
        "conf_mat_net = test_conf_mat_res_net(res_net_early, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now I try to remove both layer 3 and 4."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 187,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 2.3149  Acc: 40.6%\n",
            "200 [4]  Loss: 2.2386  Acc: 43.8%\n",
            "300 [6]  Loss: 2.0212  Acc: 56.2%\n",
            "400 [8]  Loss: 2.0070  Acc: 53.1%\n",
            "500 [10]  Loss: 1.6248  Acc: 65.6%\n",
            "600 [13]  Loss: 1.6581  Acc: 71.9%\n",
            "700 [15]  Loss: 1.8077  Acc: 53.1%\n",
            "800 [17]  Loss: 1.4594  Acc: 71.9%\n",
            "900 [19]  Loss: 1.3478  Acc: 78.1%\n",
            "1000 [21]  Loss: 1.3903  Acc: 71.9%\n",
            "1100 [23]  Loss: 1.4775  Acc: 68.8%\n",
            "1200 [26]  Loss: 1.5306  Acc: 56.2%\n",
            "1300 [28]  Loss: 1.2425  Acc: 65.6%\n",
            "1400 [30]  Loss: 1.1928  Acc: 68.8%\n",
            "1500 [32]  Loss: 1.1539  Acc: 68.8%\n",
            "1600 [34]  Loss: 1.4643  Acc: 53.1%\n",
            "1700 [36]  Loss: 1.2484  Acc: 65.6%\n",
            "1800 [39]  Loss: 1.1940  Acc: 59.4%\n",
            "1900 [41]  Loss: 0.9070  Acc: 81.2%\n",
            "2000 [43]  Loss: 1.1720  Acc: 71.9%\n",
            "2100 [45]  Loss: 0.9848  Acc: 78.1%\n",
            "2200 [47]  Loss: 1.0442  Acc: 78.1%\n",
            "2300 [49]  Loss: 1.0212  Acc: 68.8%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 71636 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 71.0%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 65636 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "res_net_very_early = torchvision.models.resnet18(weights=torchvision.models.ResNet18_Weights.DEFAULT) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "\n",
        "# replacing layer 3 and 4 with Identity\n",
        "res_net_very_early.layer3 = torch.nn.Identity()\n",
        "res_net_very_early.layer4 = torch.nn.Identity()\n",
        "\n",
        "# 128 are the in_features remaining after layer 2\n",
        "res_net_very_early.fc = torch.nn.Linear(128, len(train_set_net.classes))\n",
        "res_net_very_early = res_net_very_early.to(DEVICE)\n",
        "\n",
        "opti_res_net_very_early = torch.optim.Adam(res_net_very_early.fc.parameters(), lr=0.01)\n",
        "\n",
        "train_net(res_net_very_early, opti_res_net_very_early, train_loader=train_loader_net, epochs=50, verbose=True, plotting=True)\n",
        "conf_mat_net = test_conf_mat_res_net(res_net_very_early, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Fine tuning\n",
        "\n",
        "Now on the first net obtained (replacing only fc) is applied some fine tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 188,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 0.1854  Acc: 96.9%\n",
            "200 [4]  Loss: 0.3723  Acc: 87.5%\n",
            "300 [6]  Loss: 0.0576  Acc: 96.9%\n",
            "400 [8]  Loss: 0.0841  Acc: 96.9%\n",
            "500 [10]  Loss: 0.0847  Acc: 100.0%\n",
            "600 [13]  Loss: 0.0279  Acc: 100.0%\n",
            "700 [15]  Loss: 0.0186  Acc: 100.0%\n",
            "800 [17]  Loss: 0.0633  Acc: 96.9%\n",
            "900 [19]  Loss: 0.0192  Acc: 100.0%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 55672 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 0.0007  Acc: 100.0%\n",
            "200 [4]  Loss: 0.0198  Acc: 100.0%\n",
            "300 [6]  Loss: 0.0009  Acc: 100.0%\n",
            "400 [8]  Loss: 0.0008  Acc: 100.0%\n",
            "500 [10]  Loss: 0.0006  Acc: 100.0%\n",
            "600 [13]  Loss: 0.0002  Acc: 100.0%\n",
            "700 [15]  Loss: 0.0006  Acc: 100.0%\n",
            "800 [17]  Loss: 0.0015  Acc: 100.0%\n",
            "900 [19]  Loss: 0.0026  Acc: 100.0%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 53496 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 92.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66444 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "res_net = torchvision.models.resnet18(weights=weights) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "\n",
        "# 512 are the res_net.fc.in_features\n",
        "res_net.fc = torch.nn.Linear(512, len(train_set_net.classes))\n",
        "res_net = res_net.to(DEVICE)\n",
        "\n",
        "# Freeze all layers except fc and train only fc\n",
        "for param in res_net.parameters():\n",
        "    param.requires_grad = False\n",
        "for param in res_net.fc.parameters():\n",
        "    param.requires_grad = True\n",
        "\n",
        "# train only the fully connected layer\n",
        "opti_res_net_fc = torch.optim.Adam(res_net.fc.parameters(), lr=0.01)\n",
        "train_net(res_net, opti_res_net_fc, train_loader=train_loader_net, epochs=20, verbose=True, plotting=True)\n",
        "\n",
        "# Unfreeze all layers and fine-tune the whole network with a lower learning rate\n",
        "for param in res_net.parameters():\n",
        "    param.requires_grad = True\n",
        "\n",
        "opti_res_net_all = torch.optim.Adam(res_net.parameters(), lr=0.00001)\n",
        "train_net(res_net, opti_res_net_all, train_loader=train_loader_net, epochs=20, verbose=True, plotting=True)\n",
        "conf_mat_net = test_conf_mat_res_net(res_net, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's try to fine-tune on the network with layer 4 removed"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 189,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 1.8156  Acc: 62.5%\n",
            "200 [4]  Loss: 1.4552  Acc: 71.9%\n",
            "300 [6]  Loss: 1.3127  Acc: 71.9%\n",
            "400 [8]  Loss: 0.9612  Acc: 75.0%\n",
            "500 [10]  Loss: 0.7853  Acc: 87.5%\n",
            "600 [13]  Loss: 0.8216  Acc: 84.4%\n",
            "700 [15]  Loss: 0.7900  Acc: 87.5%\n",
            "800 [17]  Loss: 0.7406  Acc: 81.2%\n",
            "900 [19]  Loss: 0.5733  Acc: 84.4%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 53884 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2]  Loss: 0.6928  Acc: 81.2%\n",
            "200 [4]  Loss: 0.3026  Acc: 96.9%\n",
            "300 [6]  Loss: 0.4400  Acc: 87.5%\n",
            "400 [8]  Loss: 0.3982  Acc: 87.5%\n",
            "500 [10]  Loss: 0.3325  Acc: 90.6%\n",
            "600 [13]  Loss: 0.2232  Acc: 96.9%\n",
            "700 [15]  Loss: 0.2474  Acc: 93.8%\n",
            "800 [17]  Loss: 0.1434  Acc: 96.9%\n",
            "900 [19]  Loss: 0.1958  Acc: 100.0%\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 58704 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 90.3%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 66460 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "res_net_early = torchvision.models.resnet18(weights=weights) # equivalent to pretrained=True, but pretrained=True is deprecated\n",
        "\n",
        "# replacing layer 4 with Identity\n",
        "res_net_early.layer4 = torch.nn.Identity()\n",
        "\n",
        "res_net_early.fc = torch.nn.Linear(256, len(train_set_net.classes))\n",
        "res_net_early = res_net_early.to(DEVICE)\n",
        "\n",
        "# Freeze all layers except fc and train only fc\n",
        "for param in res_net_early.parameters():\n",
        "    param.requires_grad = False\n",
        "for param in res_net_early.fc.parameters():\n",
        "    param.requires_grad = True\n",
        "\n",
        "# train only the fully connected layer\n",
        "opti_res_net_early_fc = torch.optim.Adam(res_net_early.fc.parameters(), lr=0.01)\n",
        "train_net(res_net_early, opti_res_net_early_fc, train_loader=train_loader_net, epochs=20, verbose=True, plotting=True)\n",
        "\n",
        "# Unfreeze all layers and fine-tune the whole network with a lower learning rate\n",
        "for param in res_net_early.parameters():\n",
        "    param.requires_grad = True\n",
        "\n",
        "opti_res_net_early_all = torch.optim.Adam(res_net_early.parameters(), lr=0.00001)\n",
        "train_net(res_net_early, opti_res_net_early_all, train_loader=train_loader_net, epochs=20, verbose=True, plotting=True)\n",
        "conf_mat_net = test_conf_mat_res_net(res_net_early, test_loader_net, nclasses=len(train_set_net.classes), verbose=True)\n",
        "plot_conf_mat(conf_mat_net, train_set_net.classes)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "lab",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
