{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2fWDjZRP5GZ"
      },
      "source": [
        "# Cake classification (part 1)\n",
        "\n",
        "Image recognition is a very important application of machine learning.\n",
        "There are two main approaches:\n",
        "1. with hand-crafted image features processed by a classification\n",
        "  model;\n",
        "2. with deep-learning, with a neural network that directly\n",
        "  processes the image pixels.  Convolutional neural networks (CNN) are\n",
        "  the most suitable architecture for this application.\n",
        "\n",
        "## Lab activity\n",
        "\n",
        "In this exercise we will build classifiers for the classification of\n",
        "images of cakes.\n",
        "\n",
        "The data set contains 120 images for each of 15 kinds of cake.  For\n",
        "each class 100 images are in the training set and 20 form the test\n",
        "set.  All the images have been resized to $224 \\times 224$ pixels."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uMZMlqDSc15e"
      },
      "outputs": [],
      "source": [
        "# !pip install gdown\n",
        "URL = 'https://drive.google.com/file/d/1_sixkDSQEN6TYyLVmXXmTNFF_9SIiFjx/view?usp=sharing'\n",
        "!gdown --fuzzy $URL -O- | tar -xz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "98dnoISnRooW"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "Ws62ZJ0pRgip"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['chocolate_cake', 'waffles', 'cup_cakes', 'macarons', 'donuts', 'red_velvet_cake', 'panna_cotta', 'chocolate_mousse', 'chocolate_cake', 'apple_pie', 'red_velvet_cake', 'cup_cakes', 'chocolate_cake', 'cannoli', 'cannoli', 'churros', 'ice_cream', 'cannoli', 'tiramisu', 'panna_cotta', 'donuts', 'churros', 'panna_cotta', 'waffles', 'churros', 'cannoli', 'carrot_cake', 'waffles', 'chocolate_mousse', 'creme_brulee', 'chocolate_mousse', 'ice_cream']\n",
            "torch.Size([32, 3, 224, 224])\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 1314196 characters]",
            "text/plain": [
              "<Figure size 1280x960 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "transform = torchvision.transforms.Compose([\n",
        "    torchvision.transforms.RandomHorizontalFlip(),\n",
        "    torchvision.transforms.ToTensor()\n",
        "])\n",
        "\n",
        "train_set = torchvision.datasets.ImageFolder(\"./cake-classification/images/train\", transform=transform)\n",
        "train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True, drop_last=True, num_workers=2)\n",
        "\n",
        "test_set = torchvision.datasets.ImageFolder(\"./cake-classification/images/test\", transform=torchvision.transforms.ToTensor())\n",
        "test_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=False, drop_last=False, num_workers=2)\n",
        "\n",
        "\n",
        "images, labels = next(iter(train_loader))\n",
        "plt.figure(dpi=200)\n",
        "grid = torchvision.utils.make_grid(images, nrow=8, normalize=True).permute(1, 2, 0)\n",
        "plt.axis(\"off\")\n",
        "plt.imshow(grid)\n",
        "print([train_set.classes[i] for i in labels])\n",
        "print(images.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjMTPP9X_d56"
      },
      "source": [
        "## Features\n",
        "\n",
        "For feature extraction we will use low-level features:\n",
        "1. color histograms;\n",
        "2. edge direction histograms;\n",
        "3. cooccurence matrices.\n",
        "\n",
        "These can be easily implemented in pytorch by exploiting the `torch.bincount` function, which builds histograms of integers between zero and a given maximum. There are a few tricks to make the computation moreefficient; for instance, by manipulating the integers so that a batch of images can be processed without loops, and with a single call to `torch.bincount`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uy39vGgvP3qg"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([7, 729])\n"
          ]
        }
      ],
      "source": [
        "def color_histogram(img, bins=64):\n",
        "    n = img.shape[0]\n",
        "    img = (img * bins).long().clamp(0, bins - 1)\n",
        "    img = img + torch.arange(0, 3 * n * bins, bins, device=img.device).view(n, 3, 1, 1)\n",
        "    hist = torch.bincount(img.view(-1), minlength=(3 * n * bins)).view(n, 3 * bins)\n",
        "    return hist / img[0, 1].numel()\n",
        "\n",
        "\n",
        "def edge_direction_histogram(img, bins=64):\n",
        "    n = img.shape[0]\n",
        "    SOBEL = torch.tensor([\n",
        "        [[[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]],\n",
        "        [[[-1, -2, -1], [0, 0, 0], [1, 2, 1]]]\n",
        "    ], dtype=img.dtype, device=img.device)\n",
        "    gray = img.mean(1, True)\n",
        "    grad = torch.nn.functional.conv2d(gray, SOBEL)\n",
        "    magnitude = grad.norm(dim=1)\n",
        "    angle = torch.atan2(grad[:, 0], grad[:, 1])\n",
        "    directions = (bins * angle / (2 * torch.pi)).long() % bins\n",
        "    directions = directions + torch.arange(0, n * bins, bins, device=img.device).view(n, 1, 1)\n",
        "    hist = torch.bincount(directions.view(-1), weights=magnitude.view(-1), minlength=n * bins).view(n, bins)\n",
        "    hist = hist / hist.sum(1, True).clamp(1e-6)\n",
        "    return hist\n",
        "\n",
        "\n",
        "def _cooccurrence_matrix_dir(values, bins, di, dj):\n",
        "    b, m, n = values.shape\n",
        "    codes = values[:, :m - di, :n - dj] + bins * values[:, di:, dj:]\n",
        "    codes += torch.arange(0, b * (bins ** 2), bins ** 2, device=values.device).view(b, 1, 1)\n",
        "    entries = torch.bincount(codes.view(-1), minlength=b * (bins ** 2)).view(b, bins, bins)\n",
        "    return entries\n",
        "\n",
        "\n",
        "def rgb_cooccurrence_matrix(img, quantization=3, distance=10):\n",
        "    n = img.shape[0]\n",
        "    img = (img * quantization).long().clamp(0, quantization - 1)\n",
        "    img = (img * (torch.arange(3, device=img.device) ** quantization).view(1, 3, 1, 1)).sum(1)\n",
        "    bins = quantization ** 3            # the num of features is actually the square of this cause i do 2 mats\n",
        "    mat = _cooccurrence_matrix_dir(img, bins, distance, 0)\n",
        "    mat += _cooccurrence_matrix_dir(img, bins, 0, distance)\n",
        "    mat = (mat + mat.permute(0, 2, 1)).view(n, -1)\n",
        "    mat = mat / mat.sum(1, True)\n",
        "    return mat\n",
        "\n",
        "\n",
        "x = torch.rand(7, 3, 200, 200)\n",
        "h = rgb_cooccurrence_matrix(x)\n",
        "print(h.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfGCbAHLAVd3"
      },
      "source": [
        "## Classifier\n",
        "\n",
        "Let's start with logistic regression. A more sophisticated (e.g. a MLP) could be used as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "6SqpcCHSitCA"
      },
      "outputs": [],
      "source": [
        "LR = 0.01\n",
        "DEVICE = (\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "model = torch.nn.Linear(192, len(train_set.classes))            # the histo has 64 bins, so for 3 channels you have 192 features\n",
        "model.to(DEVICE)\n",
        "\n",
        "# qui usiamo adam invece di SGD, la main difference è che adam ha una sorta di normalizzaione automatica\n",
        "# no momentum, in generale più facile da usare\n",
        "# typically dovrebbe ottenere la stessa performance di sgd ma senza bisogno di beccare il parametro esatto (non cambia troppo con lr diversi)\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=LR)     \n",
        "loss_fun = torch.nn.CrossEntropyLoss()              # you avoid the softmax if you use the cross entropy! You can use it later to compute probabilities"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m4Ib7zQYAhdu"
      },
      "source": [
        "## Training\n",
        "\n",
        "The training loop is organized as usual. We will need to extract features from the images and pass them to the classifier."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "pfDzNABikhIZ"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100 [2] Loss: 2.2945 Acc: 15.6\n",
            "200 [4] Loss: 2.2606 Acc: 34.4\n",
            "300 [6] Loss: 1.9722 Acc: 43.8\n",
            "400 [8] Loss: 2.2416 Acc: 34.4\n",
            "500 [10] Loss: 1.9270 Acc: 53.1\n",
            "600 [13] Loss: 2.0378 Acc: 31.2\n",
            "700 [15] Loss: 1.8322 Acc: 56.2\n",
            "800 [17] Loss: 1.7900 Acc: 53.1\n",
            "900 [19] Loss: 2.1048 Acc: 31.2\n"
          ]
        }
      ],
      "source": [
        "EPOCHS = 20\n",
        "\n",
        "# mio\n",
        "steps = 0\n",
        "for epoch in range(EPOCHS):\n",
        "    for imgs, labels in train_loader:\n",
        "        imgs = imgs.to(DEVICE)\n",
        "        labels = labels.to(DEVICE)\n",
        "        optimizer.zero_grad()\n",
        "        features = color_histogram(imgs)\n",
        "        logits = model(features)\n",
        "        loss = loss_fun(logits, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        steps += 1\n",
        "        if steps % 100 == 0:\n",
        "            predictions = logits.argmax(dim=1)\n",
        "            accuracy = (predictions == labels).float().mean()\n",
        "            print(f\"{steps} [{epoch}] Loss: {loss.item():.4f} Acc: {accuracy.item() * 100:.1f}\")\n",
        "\n",
        "        "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRuYA_CIA_Iu"
      },
      "source": [
        "## Evaluation\n",
        "\n",
        "To evaluate the model, we build a confusion matrix and use it to compute the accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "QTx8s1pdnNYE"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 35.1%\n"
          ]
        }
      ],
      "source": [
        "nclasses = len(train_set.classes)\n",
        "cm = torch.zeros(nclasses, nclasses, dtype=torch.long)\n",
        "\n",
        "\n",
        "for imgs, labels in test_loader:\n",
        "    features = color_histogram(imgs.to(DEVICE))\n",
        "    labels = labels.to(DEVICE)\n",
        "    with torch.no_grad():\n",
        "        logits = model(features)\n",
        "    predictions = torch.argmax(logits, dim=1)\n",
        "\n",
        "    codes = labels * nclasses + predictions\n",
        "    hist = torch.bincount(codes.cpu(), minlength=nclasses ** 2)\n",
        "    cm += hist.view(nclasses, nclasses)\n",
        "    \n",
        "\n",
        "\n",
        "accuracy = cm.diag().sum() / cm.sum()\n",
        "print(f\"Accuracy: {100 * accuracy:.1f}%\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BTyZqTeBMoA"
      },
      "source": [
        "Display the confusion matrix."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "krujBRrXn8Z-"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<matplotlib.colorbar.Colorbar at 0x7f8c2faa46b0>"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 67796 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "cmnorm = cm / cm.sum(1)\n",
        "\n",
        "plt.figure(dpi=100)\n",
        "plt.imshow(cm, cmap=\"Blues\")\n",
        "plt.xticks(range(nclasses), train_set.classes, rotation=90)\n",
        "plt.yticks(range(nclasses), train_set.classes)\n",
        "plt.colorbar()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [],
      "source": [
        "# possiamo complicare sul classifier (al posto che un solo layer lineare aggiungere, fare una piccola net)\n",
        "# posso concatenare differenti features su un solo grande feature vector (color histo, rgb_concurr ecc)\n",
        "# non mi aspetto grossi risultati tho\n",
        "# se uso solo edge histo mi aspetto nulla di che, dato che non uso affatto il colore. Concurr mi aspetto in between"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "lab",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
