{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JipsEcH-oQsI"
      },
      "source": [
        "# Speech commands\n",
        "\n",
        "*Speech recognition* is a task in which the goal is to infer the words expressed by a speaker from a voice recording.  The input is usually just the audio signal as recorded by a microphone (mono or stereo).  The output is the corresponding sequence of words.\n",
        "\n",
        "Speech recognition systems are usually required to be able to transcribe any sequence of meaningful words.  However, there are applications in which a single word at a time is expected, and only a small set of words are of interest.  As an extreme example, in *trigger word detection* only a specific word, or a short sentence need to be recognized (\"Alexa!\", or \"OK Google!\").\n",
        "\n",
        "\n",
        "## Lab Activity\n",
        "\n",
        "Here we will focus on a particular case of of speech recognition, in which the audio represents the pronunciation of a single word taken from a list of 35 commands. It is a classification problem.\n",
        "\n",
        "Multilayer perceptrons will be used to classify the data.\n",
        "\n",
        "### Dataset\n",
        "In this lab exercise the [Speech Commands Data Set](https://www.tensorflow.org/datasets/catalog/speech_commands) will be used.\n",
        "\n",
        "The data set includes 105829 recordings of the 35 words uttered by many different speakers. It has been divided into a training set of 84291 audio clips, a validation set of 12162 and a test set including 9376 clips. Recordings have a single channel (mono) at a frequency of 16 KHz."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code\n",
        "All the code from now on is organized to be reusable for later experiments with different parameters.\n",
        "\n",
        "Imports:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XvqmKGjRJey5"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchaudio\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib import cm\n",
        "import IPython.display as ipd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 194,
      "metadata": {
        "id": "Ovh5_8VAokjx"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "cuda\n"
          ]
        }
      ],
      "source": [
        "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "print(DEVICE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Dataset\n",
        "Dataset is loaded and an example is visualized"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 195,
      "metadata": {
        "id": "faqcalX9olZN"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "84843 11005\n",
            "torch.Size([1, 16000]) 16000 yes 8281a2a8 2\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7fc46577bc50>]"
            ]
          },
          "execution_count": 195,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "[encoded data, 30544 characters]",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "train_set = torchaudio.datasets.SPEECHCOMMANDS(\".\", download=True, subset=\"training\")\n",
        "test_set = torchaudio.datasets.SPEECHCOMMANDS(\".\", subset=\"testing\")\n",
        "\n",
        "print(len(train_set), len(test_set))\n",
        "\n",
        "# load one example\n",
        "item = train_set[80010]\n",
        "waveform = item[0]\n",
        "sample_rate = item[1]\n",
        "command = item[2]\n",
        "speaker_id = item[3]\n",
        "utterance_number = item[4]\n",
        "print(waveform.shape, sample_rate, command, speaker_id, utterance_number)\n",
        "\n",
        "ipd.Audio(waveform[0].numpy(), rate=sample_rate)\n",
        "plt.plot(waveform[0].numpy())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TEyFG9lRpU-C"
      },
      "source": [
        "Numerical labels are assigned to the commands"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 196,
      "metadata": {
        "id": "4XGSd_supRBy"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['backward', 'bed', 'bird', 'cat', 'dog', 'down', 'eight', 'five', 'follow', 'forward', 'four', 'go', 'happy', 'house', 'learn', 'left', 'marvin', 'nine', 'no', 'off', 'on', 'one', 'right', 'seven', 'sheila', 'six', 'stop', 'three', 'tree', 'two', 'up', 'visual', 'wow', 'yes', 'zero']\n",
            "{'backward': 0, 'bed': 1, 'bird': 2, 'cat': 3, 'dog': 4, 'down': 5, 'eight': 6, 'five': 7, 'follow': 8, 'forward': 9, 'four': 10, 'go': 11, 'happy': 12, 'house': 13, 'learn': 14, 'left': 15, 'marvin': 16, 'nine': 17, 'no': 18, 'off': 19, 'on': 20, 'one': 21, 'right': 22, 'seven': 23, 'sheila': 24, 'six': 25, 'stop': 26, 'three': 27, 'tree': 28, 'two': 29, 'up': 30, 'visual': 31, 'wow': 32, 'yes': 33, 'zero': 34}\n"
          ]
        }
      ],
      "source": [
        "# List of the commands and a dictionary to map the commands to indices \n",
        "# test set is used because it's smaller -> faster\n",
        "commands = sorted(set(item[2] for item in test_set))\n",
        "command_index = {commands[i]: i for i in range(len(commands))}\n",
        "print(commands)\n",
        "print(command_index)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5XvjystOwNa"
      },
      "source": [
        "### Features extraction\n",
        "Feature extraction is performed later by computing the spectrogram of the audio track. Here an example is shown"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 197,
      "metadata": {
        "id": "o0h8dWM3sk43"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "torch.Size([1, 32, 32])\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 18492 characters]",
            "text/plain": [
              "<Figure size 768x576 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "backward\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "execution_count": 197,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "n_fft = 1024  # Size of FFT windows (in realtà avremo il doppio di windows perché di default si muove di mezza windows)\n",
        "n_mels = 32   # Number of filters\n",
        "sample_rate = 16000\n",
        "\n",
        "mel_spectrogram = torchaudio.transforms.MelSpectrogram(\n",
        "    sample_rate=sample_rate,\n",
        "    n_fft=n_fft,\n",
        "    n_mels=n_mels,\n",
        "    normalized=True)\n",
        "\n",
        "# Show one example.\n",
        "index = 1100\n",
        "features = mel_spectrogram(train_set[index][0])\n",
        "print(features.shape)\n",
        "plt.figure(dpi=120)\n",
        "plt.imshow(features[0].cpu(), cmap=\"hot\")\n",
        "plt.colorbar()\n",
        "plt.show()\n",
        "print(train_set[index][2])\n",
        "ipd.Audio(train_set[index][0].numpy(), rate=sample_rate)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9PM3oOtDR7zK"
      },
      "source": [
        "### Data loaders\n",
        "\n",
        "Data loaders are used to manage samples. Since the original collate function caused the spectrograms to be retrieved again every time a different model was trained, it has not be used as it is in the next cell. Spectrograms have been pre-computed and saved: this removed the CPU bottleneck (since computations in the collate function are CPU-bound) during training and allowed the GPU to perform at its best, without being limited by data loading and preprocessing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 198,
      "metadata": {
        "id": "aqpmcE6KKgH9"
      },
      "outputs": [],
      "source": [
        "LENGTH = 32  # Lenght of the features in the time dimension.\n",
        "BATCH_SIZE = 100\n",
        "\n",
        "# the collate function returns the tensors representing the batches of data\n",
        "def collate_fn(batch):\n",
        "    features = []\n",
        "    labels = []\n",
        "    for item in batch:\n",
        "        waveform = item[0]\n",
        "        # make sure that the time lenght is always LENGTH\n",
        "        f = mel_spectrogram(waveform)\n",
        "        if f.shape[2] > LENGTH:         # truncate if too long\n",
        "            f = f[:, :, :LENGTH]\n",
        "        elif f.shape[2] < LENGTH:       # pad if too short\n",
        "            f = torch.nn.functional.pad(f, (0, LENGTH - f.shape[2]))\n",
        "        features.append(f)\n",
        "        label = command_index[item[2]]\n",
        "        labels.append(label)\n",
        "    features = torch.stack(features, 0)\n",
        "    labels = torch.tensor(labels)\n",
        "    return features, labels\n",
        "\n",
        "# the batch contains both features and labels\n",
        "# important to shuffle training\n",
        "# train_loader = torch.utils.data.DataLoader(train_set, batch_size=BATCH_SIZE, collate_fn=collate_fn, shuffle=True, num_workers=12)\n",
        "# test_loader = torch.utils.data.DataLoader(test_set, batch_size=BATCH_SIZE, collate_fn=collate_fn, shuffle=False, num_workers=12)\n",
        "\n",
        "# batch = next(iter(train_loader))\n",
        "# print(batch[0].shape, batch[1].shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 199,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Pre-compute mel spectrograms\n",
        "# this is done so that when the model is trained multiple times, \n",
        "# the collate function is not called every time (it would compute the \n",
        "# same spectrograms multiple times)\n",
        "def compute_and_save_spectrograms(dataset, filename=\"data.pt\"):\n",
        "    features = []\n",
        "    labels = []\n",
        "    for item in dataset:\n",
        "        waveform = item[0]\n",
        "        # make sure that the time lenght is always LENGTH\n",
        "        f = mel_spectrogram(waveform)\n",
        "        if f.shape[2] > LENGTH:         # truncate if too long\n",
        "            f = f[:, :, :LENGTH]\n",
        "        elif f.shape[2] < LENGTH:       # pad if too short\n",
        "            f = torch.nn.functional.pad(f, (0, LENGTH - f.shape[2]))\n",
        "        features.append(f)\n",
        "        label = command_index[item[2]]\n",
        "        labels.append(label)\n",
        "    features = torch.stack(features, 0)\n",
        "    labels = torch.tensor(labels)\n",
        "    # Save features and labels\n",
        "    torch.save({'features': features, 'labels': labels}, filename)\n",
        "\n",
        "compute_and_save_spectrograms(train_set, \"train.pt\")\n",
        "compute_and_save_spectrograms(test_set, \"test.pt\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 200,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Now, the loaders are initialized with the pre-computed spectrograms\n",
        "train_data = torch.load(\"train.pt\")\n",
        "test_data = torch.load(\"test.pt\")\n",
        "\n",
        "train_dataset_precomputed = torch.utils.data.TensorDataset(train_data['features'], train_data['labels'])\n",
        "test_dataset_precomputed = torch.utils.data.TensorDataset(test_data['features'], test_data['labels'])\n",
        "\n",
        "train_loader = torch.utils.data.DataLoader(train_dataset_precomputed, batch_size=BATCH_SIZE, shuffle=True, num_workers=12)\n",
        "test_loader = torch.utils.data.DataLoader(test_dataset_precomputed, batch_size=BATCH_SIZE, shuffle=False, num_workers=12)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJD4z68ESzBu"
      },
      "source": [
        "### Neural Networks\n",
        "\n",
        "First try is a simple mlp with 1 hidden layer with 64 neurons."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 201,
      "metadata": {
        "id": "K9JzUBbSSIRr"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n"
          ]
        }
      ],
      "source": [
        "net = torch.nn.Sequential(\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net.parameters())\n",
        "print(f\"Number of parameters: {ps}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-gsSDWvUHWA"
      },
      "source": [
        "### Training loop\n",
        "\n",
        "Stochastic gradient descent with momentum has been used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 202,
      "metadata": {},
      "outputs": [],
      "source": [
        "def train(model, train_loader, epochs=10, lr=1e-2, momentum=0.9, filename=\"speech_commands.pt\", plotting=False, verbose=True):\n",
        "    loss_fun = torch.nn.CrossEntropyLoss()\n",
        "    optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum)\n",
        "    step = 0\n",
        "\n",
        "    steps = []\n",
        "    accuracies = []\n",
        "    losses = []\n",
        "\n",
        "    for epoch in range(epochs):\n",
        "        for features, labels in train_loader:\n",
        "            features = features.to(DEVICE)\n",
        "            labels = labels.to(DEVICE)\n",
        "            optimizer.zero_grad()\n",
        "            logits = model(features)\n",
        "            loss = loss_fun(logits, labels)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            step += 1\n",
        "            if step % 1000 == 0:\n",
        "                predictions = torch.argmax(logits, dim=1)\n",
        "                accuracy = (predictions == labels).float().mean()\n",
        "                steps.append(step)\n",
        "                accuracies.append(accuracy.item())\n",
        "                losses.append(loss.item())\n",
        "                if verbose:\n",
        "                    print(f\"Epoch {epoch} step {step} loss {loss.item():.4f} acc. {accuracy * 100:.1f}%\")\n",
        "    torch.save(model.state_dict(), filename)\n",
        "\n",
        "    if plotting:\n",
        "        plt.figure(figsize=(8, 5))\n",
        "        fig, ax1 = plt.subplots(figsize=(8, 5))\n",
        "        set1 = cm.get_cmap('Set1')\n",
        "        color = set1(2)\n",
        "        ax1.set_xlabel(\"Iterations\")\n",
        "        ax1.set_ylabel(\"Accuracy\", color=color,  fontsize=14)\n",
        "        ax1.plot(steps, accuracies, label=\"Accuracy\", color=color, linewidth=2)\n",
        "        ax1.tick_params(axis='y', labelcolor=color)\n",
        "        ax1.set_ylim(0, 1)  # accuracy between 0 and 1\n",
        "\n",
        "        ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis\n",
        "\n",
        "        color = set1(1)\n",
        "        ax2.set_ylabel(\"Loss\", color=color,  fontsize=14)\n",
        "        ax2.plot(steps, losses, label=\"Loss\", color=color, linewidth=2)\n",
        "        ax2.tick_params(axis='y', labelcolor=color)\n",
        "\n",
        "        fig.tight_layout()\n",
        "        plt.title(\"Accuracy and Loss over Iterations\")\n",
        "        plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 203,
      "metadata": {
        "id": "R9W2VNmBSjPi"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1 step 1000 loss 3.1421 acc. 11.0%\n",
            "Epoch 2 step 2000 loss 2.6750 acc. 28.0%\n",
            "Epoch 3 step 3000 loss 2.6697 acc. 33.0%\n",
            "Epoch 4 step 4000 loss 2.4129 acc. 35.0%\n",
            "Epoch 5 step 5000 loss 2.1823 acc. 34.0%\n",
            "Epoch 7 step 6000 loss 2.2090 acc. 40.0%\n",
            "Epoch 8 step 7000 loss 2.0406 acc. 52.0%\n",
            "Epoch 9 step 8000 loss 2.1857 acc. 36.0%\n",
            "Epoch 10 step 9000 loss 2.2129 acc. 43.0%\n",
            "Epoch 11 step 10000 loss 2.1103 acc. 40.0%\n",
            "Epoch 12 step 11000 loss 2.5016 acc. 28.0%\n",
            "Epoch 14 step 12000 loss 1.7477 acc. 50.0%\n",
            "Epoch 15 step 13000 loss 2.3369 acc. 45.0%\n",
            "Epoch 16 step 14000 loss 2.1151 acc. 42.0%\n",
            "Epoch 17 step 15000 loss 1.8046 acc. 44.0%\n",
            "Epoch 18 step 16000 loss 2.1453 acc. 40.0%\n",
            "Epoch 20 step 17000 loss 1.9101 acc. 53.0%\n",
            "Epoch 21 step 18000 loss 1.9978 acc. 40.0%\n",
            "Epoch 22 step 19000 loss 2.0127 acc. 50.0%\n",
            "Epoch 23 step 20000 loss 1.8793 acc. 48.0%\n",
            "Epoch 24 step 21000 loss 1.9755 acc. 47.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 75860 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "train(net, train_loader, epochs=25, lr=1e-2, momentum=0.9, plotting=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5NKkePRZUVtN"
      },
      "source": [
        "### Evaluation (test set)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 204,
      "metadata": {
        "id": "ddLLGaTgVcPU"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 38.5%\n"
          ]
        }
      ],
      "source": [
        "def test(model, test_loader):\n",
        "    ok = 0\n",
        "    tot = 0\n",
        "\n",
        "    for features, labels in test_loader:\n",
        "        features = features.to(DEVICE)\n",
        "        labels = labels.to(DEVICE)\n",
        "        with torch.no_grad():       # no grad used because in test no backpropagation\n",
        "            logits = model(features)\n",
        "        predictions = torch.argmax(logits, dim=1)\n",
        "        ok += (predictions == labels).long().sum()\n",
        "        tot += len(labels)\n",
        "\n",
        "    print(f\"Accuracy: {(ok / tot) * 100:.1f}%\")\n",
        "\n",
        "test(net, test_loader)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2xKf07jUo0l"
      },
      "source": [
        "### Confusion matrix\n",
        "\n",
        "A normalized confusion matrix is computed and plotted"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 205,
      "metadata": {
        "id": "IWNytn3-Uf_V"
      },
      "outputs": [],
      "source": [
        "def make_confusion_matrix(model, test_loader):\n",
        "    # Create a confusion matrix\n",
        "    cm = torch.zeros(len(commands), len(commands), dtype=int)\n",
        "\n",
        "    for features, labels in test_loader:\n",
        "        features = features.to(DEVICE)\n",
        "        labels = labels.to(DEVICE)\n",
        "        with torch.no_grad():           # no grad used because in test no backpropagation\n",
        "            logits = model(features)\n",
        "        predictions = torch.argmax(logits, dim=1)\n",
        "    \n",
        "        # Basically I have a unique code for each pair (label, prediction)\n",
        "        # Then with bincount I count the occurences of the particular pair\n",
        "        # Finally with hist.view I reshape to be a square matrix\n",
        "        codes = labels * len(commands) + predictions\n",
        "        hist = torch.bincount(codes.cpu(), minlength=len(commands) ** 2)\n",
        "        cm += hist.view(len(commands), len(commands))\n",
        "\n",
        "    cm = cm / cm.sum(1).unsqueeze(1)  # Normalization\n",
        "    return cm\n",
        "\n",
        "def plot_confusion_matrix(model, test_loader):\n",
        "    cm = make_confusion_matrix(model, test_loader)\n",
        "    plt.figure(figsize=(13,13), dpi=300)\n",
        "    plt.imshow(cm.numpy(), cmap=\"Blues\")\n",
        "    plt.xticks(range(len(commands)), commands, rotation=90)\n",
        "    plt.yticks(range(len(commands)), commands)\n",
        "    plt.title(\"Confusion matrix\")\n",
        "    plt.colorbar()\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 206,
      "metadata": {
        "id": "eQ85CmOCWNSt"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 341820 characters]",
            "text/plain": [
              "<Figure size 3900x3900 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "plot_confusion_matrix(net, test_loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8kkxmnV2TSGp"
      },
      "source": [
        "### Normalization\n",
        "\n",
        "Intance normalization 2D is used firts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 207,
      "metadata": {
        "id": "Q-RBlFKLTQnV"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n"
          ]
        }
      ],
      "source": [
        "# Include instance normalization in the definition of the network and retrain it.\n",
        "\n",
        "net_normalized_instance = torch.nn.Sequential(\n",
        "    torch.nn.InstanceNorm2d(1), \n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_instance.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_instance.parameters())\n",
        "print(f\"Number of parameters: {ps}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 208,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1 step 1000 loss 2.1218 acc. 39.0%\n",
            "Epoch 2 step 2000 loss 1.8302 acc. 43.0%\n",
            "Epoch 3 step 3000 loss 1.6860 acc. 49.0%\n",
            "Epoch 4 step 4000 loss 1.7092 acc. 50.0%\n",
            "Epoch 5 step 5000 loss 1.6239 acc. 50.0%\n",
            "Epoch 7 step 6000 loss 1.3857 acc. 54.0%\n",
            "Epoch 8 step 7000 loss 1.4839 acc. 51.0%\n",
            "Epoch 9 step 8000 loss 1.9359 acc. 49.0%\n",
            "Epoch 10 step 9000 loss 1.3219 acc. 67.0%\n",
            "Epoch 11 step 10000 loss 1.4949 acc. 54.0%\n",
            "Epoch 12 step 11000 loss 1.2436 acc. 63.0%\n",
            "Epoch 14 step 12000 loss 1.2910 acc. 58.0%\n",
            "Epoch 15 step 13000 loss 1.2989 acc. 59.0%\n",
            "Epoch 16 step 14000 loss 1.2283 acc. 55.0%\n",
            "Epoch 17 step 15000 loss 1.2476 acc. 59.0%\n",
            "Epoch 18 step 16000 loss 1.2359 acc. 56.0%\n",
            "Epoch 20 step 17000 loss 1.2603 acc. 63.0%\n",
            "Epoch 21 step 18000 loss 1.2543 acc. 60.0%\n",
            "Epoch 22 step 19000 loss 1.3725 acc. 63.0%\n",
            "Epoch 23 step 20000 loss 1.1498 acc. 62.0%\n",
            "Epoch 24 step 21000 loss 1.2312 acc. 63.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 67568 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "train(net_normalized_instance, train_loader, epochs=25, lr=1e-3, momentum=0.99, plotting=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 209,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 51.6%\n"
          ]
        }
      ],
      "source": [
        "test(net_normalized_instance, test_loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_paGN1ESd-Dx"
      },
      "source": [
        "## Feature normalization comparison\n",
        "\n",
        "The instance normalization used before is compared with: \n",
        "* L2 and L1 instance normalizations\n",
        "* mean-var global normalization\n",
        "* global whitening\n",
        "* batch normalization\n",
        "\n",
        "L2 and L1 instance and batch are implemented as custom layers. Global Mean-Var and Whitening are pre-applied to the whole dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 210,
      "metadata": {
        "id": "r5UybZafYI8A"
      },
      "outputs": [],
      "source": [
        "class InstanceL1Norm(torch.nn.Module):\n",
        "    def __init__(self, eps=1e-15):\n",
        "        super().__init__()\n",
        "        self.eps = eps\n",
        "\n",
        "    def forward(self, x):\n",
        "        # x: [B, C, H, W]\n",
        "        return torch.nn.functional.normalize(x, p=1, dim=(2, 3), eps=self.eps)  # L1 normalization\n",
        "    \n",
        "class InstanceL2Norm(torch.nn.Module):\n",
        "    def __init__(self, eps=1e-15):\n",
        "        super().__init__()\n",
        "        self.eps = eps\n",
        "\n",
        "    def forward(self, x):\n",
        "        # x: [B, C, H, W]\n",
        "        return torch.nn.functional.normalize(x, p=2, dim=(2, 3), eps=self.eps)  # L2 normalization\n",
        "    \n",
        "def meanvar_norm(Xtrain, Xtest):\n",
        "    # mean and variance normalization\n",
        "    mean = Xtrain.mean(dim=(0, 2, 3), keepdim=True)\n",
        "    std = Xtrain.std(dim=(0, 2, 3), keepdim=True)\n",
        "    Xtrain = (Xtrain - mean) / std\n",
        "    Xtest = (Xtest - mean) / std\n",
        "    return Xtrain, Xtest\n",
        "\n",
        "def whitening_norm(Xtrain, Xtest):\n",
        "    # whitening normalization\n",
        "    X_flat = Xtrain.view(Xtrain.shape[0], -1)\n",
        "    X_test_flat = Xtest.view(Xtest.shape[0], -1)\n",
        "    mean = X_flat.mean(dim=0, keepdim=True)\n",
        "    sigma = torch.cov(X_flat.T)\n",
        "    evals, evecs = torch.linalg.eigh(sigma)\n",
        "    w = evecs / torch.sqrt(evals)\n",
        "    X_flat = (X_flat - mean) @ w\n",
        "    X_test_flat = (X_test_flat - mean) @ w\n",
        "    Xtrain = X_flat.view(Xtrain.shape)\n",
        "    Xtest = X_test_flat.view(Xtest.shape)\n",
        "    return Xtrain, Xtest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 211,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n",
            "Epoch 1 step 1000 loss 3.4864 acc. 9.0%\n",
            "Epoch 2 step 2000 loss 3.2452 acc. 6.0%\n",
            "Epoch 3 step 3000 loss 3.0170 acc. 13.0%\n",
            "Epoch 4 step 4000 loss 2.9918 acc. 18.0%\n",
            "Epoch 5 step 5000 loss 2.7744 acc. 12.0%\n",
            "Epoch 7 step 6000 loss 2.8221 acc. 18.0%\n",
            "Epoch 8 step 7000 loss 2.9119 acc. 12.0%\n",
            "Epoch 9 step 8000 loss 2.8196 acc. 16.0%\n",
            "Epoch 10 step 9000 loss 2.7783 acc. 15.0%\n",
            "Epoch 11 step 10000 loss 2.6376 acc. 17.0%\n",
            "Epoch 12 step 11000 loss 2.7104 acc. 25.0%\n",
            "Epoch 14 step 12000 loss 2.7030 acc. 26.0%\n",
            "Epoch 15 step 13000 loss 2.4974 acc. 28.0%\n",
            "Epoch 16 step 14000 loss 2.5537 acc. 27.0%\n",
            "Epoch 17 step 15000 loss 2.5657 acc. 23.0%\n",
            "Epoch 18 step 16000 loss 2.5633 acc. 27.0%\n",
            "Epoch 20 step 17000 loss 2.5295 acc. 26.0%\n",
            "Epoch 21 step 18000 loss 2.5243 acc. 27.0%\n",
            "Epoch 22 step 19000 loss 2.4398 acc. 22.0%\n",
            "Epoch 23 step 20000 loss 2.6359 acc. 24.0%\n",
            "Epoch 24 step 21000 loss 2.5765 acc. 19.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 62248 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 24.1%\n"
          ]
        }
      ],
      "source": [
        "# L1 Net\n",
        "net_normalized_instance_L1 = torch.nn.Sequential(\n",
        "    InstanceL1Norm(), \n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_instance_L1.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_instance_L1.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_normalized_instance_L1, train_loader, epochs=25, lr=1e-2, momentum=0.99, plotting=True)\n",
        "test(net_normalized_instance_L1, test_loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 212,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n",
            "Epoch 1 step 1000 loss 2.6500 acc. 19.0%\n",
            "Epoch 2 step 2000 loss 2.3972 acc. 34.0%\n",
            "Epoch 3 step 3000 loss 2.4855 acc. 26.0%\n",
            "Epoch 4 step 4000 loss 1.9903 acc. 40.0%\n",
            "Epoch 5 step 5000 loss 2.0622 acc. 42.0%\n",
            "Epoch 7 step 6000 loss 1.9484 acc. 50.0%\n",
            "Epoch 8 step 7000 loss 1.9338 acc. 37.0%\n",
            "Epoch 9 step 8000 loss 1.7992 acc. 46.0%\n",
            "Epoch 10 step 9000 loss 1.8310 acc. 50.0%\n",
            "Epoch 11 step 10000 loss 1.7000 acc. 48.0%\n",
            "Epoch 12 step 11000 loss 1.7240 acc. 47.0%\n",
            "Epoch 14 step 12000 loss 1.5527 acc. 54.0%\n",
            "Epoch 15 step 13000 loss 1.7413 acc. 45.0%\n",
            "Epoch 16 step 14000 loss 1.7186 acc. 51.0%\n",
            "Epoch 17 step 15000 loss 1.7685 acc. 38.0%\n",
            "Epoch 18 step 16000 loss 1.7307 acc. 55.0%\n",
            "Epoch 20 step 17000 loss 1.3108 acc. 59.0%\n",
            "Epoch 21 step 18000 loss 1.4664 acc. 54.0%\n",
            "Epoch 22 step 19000 loss 1.6385 acc. 51.0%\n",
            "Epoch 23 step 20000 loss 1.8205 acc. 45.0%\n",
            "Epoch 24 step 21000 loss 1.4781 acc. 54.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 71752 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 46.8%\n"
          ]
        }
      ],
      "source": [
        "# L2 Net\n",
        "net_normalized_instance_L2 = torch.nn.Sequential(\n",
        "    InstanceL2Norm(), \n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_instance_L2.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_instance_L2.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_normalized_instance_L2, train_loader, epochs=25, lr=1e-2, momentum=0.99, plotting=True)\n",
        "test(net_normalized_instance_L2, test_loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 213,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n",
            "Epoch 1 step 1000 loss 3.0302 acc. 21.0%\n",
            "Epoch 2 step 2000 loss 2.7794 acc. 29.0%\n",
            "Epoch 3 step 3000 loss 2.6397 acc. 28.0%\n",
            "Epoch 4 step 4000 loss 2.6141 acc. 25.0%\n",
            "Epoch 5 step 5000 loss 2.2515 acc. 30.0%\n",
            "Epoch 7 step 6000 loss 1.9443 acc. 47.0%\n",
            "Epoch 8 step 7000 loss 2.0959 acc. 40.0%\n",
            "Epoch 9 step 8000 loss 2.1927 acc. 37.0%\n",
            "Epoch 10 step 9000 loss 2.3108 acc. 33.0%\n",
            "Epoch 11 step 10000 loss 2.3668 acc. 36.0%\n",
            "Epoch 12 step 11000 loss 2.0142 acc. 41.0%\n",
            "Epoch 14 step 12000 loss 2.2369 acc. 33.0%\n",
            "Epoch 15 step 13000 loss 1.9050 acc. 43.0%\n",
            "Epoch 16 step 14000 loss 2.0626 acc. 41.0%\n",
            "Epoch 17 step 15000 loss 1.7619 acc. 53.0%\n",
            "Epoch 18 step 16000 loss 2.0683 acc. 44.0%\n",
            "Epoch 20 step 17000 loss 1.9671 acc. 43.0%\n",
            "Epoch 21 step 18000 loss 1.8841 acc. 42.0%\n",
            "Epoch 22 step 19000 loss 2.1590 acc. 40.0%\n",
            "Epoch 23 step 20000 loss 1.7486 acc. 48.0%\n",
            "Epoch 24 step 21000 loss 1.8728 acc. 43.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 73912 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 38.9%\n"
          ]
        }
      ],
      "source": [
        "# To test the mean_var, the whole dataset has to be pre-processed\n",
        "meanvar_train, meanvar_test = meanvar_norm(train_data['features'], test_data['features'])\n",
        "train_dataset_meanvar = torch.utils.data.TensorDataset(meanvar_train, train_data['labels'])\n",
        "test_dataset_meanvar = torch.utils.data.TensorDataset(meanvar_test, test_data['labels'])\n",
        "\n",
        "train_loader_meanvar = torch.utils.data.DataLoader(train_dataset_meanvar, batch_size=BATCH_SIZE, shuffle=True, num_workers=12)\n",
        "test_loader_meanvar = torch.utils.data.DataLoader(test_dataset_meanvar, batch_size=BATCH_SIZE, shuffle=False, num_workers=12)\n",
        "\n",
        "# meanvar Net\n",
        "net_normalized_meanvar = torch.nn.Sequential(\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_meanvar.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_meanvar.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_normalized_meanvar, train_loader_meanvar, epochs=25, lr=1e-2, momentum=0.9, plotting=True)\n",
        "test(net_normalized_meanvar, test_loader_meanvar)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 214,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67875\n",
            "Epoch 1 step 1000 loss 3.1211 acc. 16.0%\n",
            "Epoch 2 step 2000 loss 2.9579 acc. 17.0%\n",
            "Epoch 3 step 3000 loss 2.7251 acc. 31.0%\n",
            "Epoch 4 step 4000 loss 2.6476 acc. 27.0%\n",
            "Epoch 5 step 5000 loss 2.3082 acc. 38.0%\n",
            "Epoch 7 step 6000 loss 2.4677 acc. 27.0%\n",
            "Epoch 8 step 7000 loss 2.3526 acc. 33.0%\n",
            "Epoch 9 step 8000 loss 2.2492 acc. 38.0%\n",
            "Epoch 10 step 9000 loss 2.2576 acc. 28.0%\n",
            "Epoch 11 step 10000 loss 2.6901 acc. 27.0%\n",
            "Epoch 12 step 11000 loss 2.3440 acc. 35.0%\n",
            "Epoch 14 step 12000 loss 1.9669 acc. 39.0%\n",
            "Epoch 15 step 13000 loss 2.2967 acc. 38.0%\n",
            "Epoch 16 step 14000 loss 2.0633 acc. 47.0%\n",
            "Epoch 17 step 15000 loss 2.1136 acc. 43.0%\n",
            "Epoch 18 step 16000 loss 1.9892 acc. 46.0%\n",
            "Epoch 20 step 17000 loss 2.0871 acc. 42.0%\n",
            "Epoch 21 step 18000 loss 1.9362 acc. 43.0%\n",
            "Epoch 22 step 19000 loss 2.0135 acc. 43.0%\n",
            "Epoch 23 step 20000 loss 2.0212 acc. 44.0%\n",
            "Epoch 24 step 21000 loss 1.8903 acc. 44.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 69532 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 28.3%\n"
          ]
        }
      ],
      "source": [
        "# Whithening is applied in a similar way to whitening normalization\n",
        "whitening_train, whitening_test = whitening_norm(train_data['features'], test_data['features'])\n",
        "train_dataset_whitening = torch.utils.data.TensorDataset(whitening_train, train_data['labels'])\n",
        "test_dataset_whitening = torch.utils.data.TensorDataset(whitening_test, test_data['labels'])\n",
        "\n",
        "train_loader_whitening = torch.utils.data.DataLoader(train_dataset_whitening, batch_size=BATCH_SIZE, shuffle=True, num_workers=12)\n",
        "test_loader_whitening = torch.utils.data.DataLoader(test_dataset_whitening, batch_size=BATCH_SIZE, shuffle=False, num_workers=12)\n",
        "\n",
        "# meanvar Net\n",
        "net_normalized_whitening = torch.nn.Sequential(\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_whitening.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_whitening.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_normalized_whitening, train_loader_whitening, epochs=25, lr=1e-2, momentum=0.9, plotting=True)\n",
        "test(net_normalized_whitening, test_loader_whitening)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 215,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 67877\n",
            "Epoch 1 step 1000 loss 2.9251 acc. 25.0%\n",
            "Epoch 2 step 2000 loss 2.8564 acc. 20.0%\n",
            "Epoch 3 step 3000 loss 2.6055 acc. 31.0%\n",
            "Epoch 4 step 4000 loss 2.3359 acc. 34.0%\n",
            "Epoch 5 step 5000 loss 2.3776 acc. 32.0%\n",
            "Epoch 7 step 6000 loss 2.1786 acc. 35.0%\n",
            "Epoch 8 step 7000 loss 1.9770 acc. 41.0%\n",
            "Epoch 9 step 8000 loss 2.1431 acc. 41.0%\n",
            "Epoch 10 step 9000 loss 2.3777 acc. 32.0%\n",
            "Epoch 11 step 10000 loss 2.2739 acc. 36.0%\n",
            "Epoch 12 step 11000 loss 1.9858 acc. 42.0%\n",
            "Epoch 14 step 12000 loss 1.9085 acc. 53.0%\n",
            "Epoch 15 step 13000 loss 2.0460 acc. 44.0%\n",
            "Epoch 16 step 14000 loss 2.1468 acc. 39.0%\n",
            "Epoch 17 step 15000 loss 1.9970 acc. 47.0%\n",
            "Epoch 18 step 16000 loss 1.8349 acc. 47.0%\n",
            "Epoch 20 step 17000 loss 2.0664 acc. 43.0%\n",
            "Epoch 21 step 18000 loss 1.9311 acc. 49.0%\n",
            "Epoch 22 step 19000 loss 1.9792 acc. 42.0%\n",
            "Epoch 23 step 20000 loss 1.9451 acc. 44.0%\n",
            "Epoch 24 step 21000 loss 2.1732 acc. 43.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 71140 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 38.6%\n"
          ]
        }
      ],
      "source": [
        "# Finally, batch normalization\n",
        "# L1 Net\n",
        "net_normalized_batch = torch.nn.Sequential(\n",
        "    torch.nn.BatchNorm2d(1),\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_normalized_batch.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_normalized_batch.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_normalized_batch, train_loader, epochs=25, lr=1e-2, momentum=0.9, plotting=True)\n",
        "test(net_normalized_batch, test_loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6nb8tjdfYIH7"
      },
      "source": [
        "## Network architecture\n",
        "Three more architectures with increasing number of hidden layers have been tested"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 141731\n",
            "Epoch 1 step 1000 loss 3.0919 acc. 15.0%\n",
            "Epoch 2 step 2000 loss 2.8082 acc. 13.0%\n",
            "Epoch 3 step 3000 loss 2.5236 acc. 28.0%\n",
            "Epoch 4 step 4000 loss 2.2365 acc. 35.0%\n",
            "Epoch 5 step 5000 loss 2.4377 acc. 28.0%\n",
            "Epoch 7 step 6000 loss 2.0887 acc. 43.0%\n",
            "Epoch 8 step 7000 loss 1.9537 acc. 42.0%\n",
            "Epoch 9 step 8000 loss 1.7843 acc. 43.0%\n",
            "Epoch 10 step 9000 loss 2.1358 acc. 38.0%\n",
            "Epoch 11 step 10000 loss 1.7289 acc. 51.0%\n",
            "Epoch 12 step 11000 loss 1.6360 acc. 49.0%\n",
            "Epoch 14 step 12000 loss 1.6131 acc. 55.0%\n",
            "Epoch 15 step 13000 loss 1.7512 acc. 50.0%\n",
            "Epoch 16 step 14000 loss 1.3848 acc. 61.0%\n",
            "Epoch 17 step 15000 loss 1.5083 acc. 56.0%\n",
            "Epoch 18 step 16000 loss 1.1943 acc. 62.0%\n",
            "Epoch 20 step 17000 loss 1.3828 acc. 58.0%\n",
            "Epoch 21 step 18000 loss 1.2916 acc. 59.0%\n",
            "Epoch 22 step 19000 loss 1.1261 acc. 64.0%\n",
            "Epoch 23 step 20000 loss 1.3080 acc. 64.0%\n",
            "Epoch 24 step 21000 loss 1.2845 acc. 60.0%\n",
            "Epoch 25 step 22000 loss 1.4286 acc. 60.0%\n",
            "Epoch 27 step 23000 loss 1.2232 acc. 61.0%\n",
            "Epoch 28 step 24000 loss 1.1730 acc. 67.0%\n",
            "Epoch 29 step 25000 loss 1.5146 acc. 57.0%\n",
            "Epoch 30 step 26000 loss 1.4017 acc. 59.0%\n",
            "Epoch 31 step 27000 loss 1.4669 acc. 57.0%\n",
            "Epoch 32 step 28000 loss 1.2280 acc. 63.0%\n",
            "Epoch 34 step 29000 loss 1.3058 acc. 56.0%\n",
            "Epoch 35 step 30000 loss 1.0665 acc. 63.0%\n",
            "Epoch 36 step 31000 loss 1.3178 acc. 63.0%\n",
            "Epoch 37 step 32000 loss 0.7651 acc. 75.0%\n",
            "Epoch 38 step 33000 loss 1.0361 acc. 68.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 72356 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 55.4%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 343504 characters]",
            "text/plain": [
              "<Figure size 3900x3900 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# 1 more hidden layer\n",
        "net_advanced = torch.nn.Sequential(\n",
        "    torch.nn.InstanceNorm2d(1),\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_advanced.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_advanced.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_advanced, train_loader, epochs=40, lr=1e-3, momentum=0.9, plotting=True)\n",
        "test(net_advanced, test_loader)\n",
        "plot_confusion_matrix(net_advanced, test_loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 305827\n",
            "Epoch 1 step 1000 loss 3.3920 acc. 8.0%\n",
            "Epoch 2 step 2000 loss 2.9539 acc. 19.0%\n",
            "Epoch 3 step 3000 loss 2.7869 acc. 22.0%\n",
            "Epoch 4 step 4000 loss 2.7439 acc. 21.0%\n",
            "Epoch 5 step 5000 loss 2.2912 acc. 30.0%\n",
            "Epoch 7 step 6000 loss 2.3124 acc. 31.0%\n",
            "Epoch 8 step 7000 loss 2.0598 acc. 37.0%\n",
            "Epoch 9 step 8000 loss 1.9025 acc. 43.0%\n",
            "Epoch 10 step 9000 loss 1.9227 acc. 41.0%\n",
            "Epoch 11 step 10000 loss 1.8303 acc. 44.0%\n",
            "Epoch 12 step 11000 loss 1.5774 acc. 50.0%\n",
            "Epoch 14 step 12000 loss 1.5100 acc. 52.0%\n",
            "Epoch 15 step 13000 loss 1.3328 acc. 66.0%\n",
            "Epoch 16 step 14000 loss 1.6317 acc. 50.0%\n",
            "Epoch 17 step 15000 loss 1.4320 acc. 57.0%\n",
            "Epoch 18 step 16000 loss 1.5996 acc. 52.0%\n",
            "Epoch 20 step 17000 loss 1.2727 acc. 65.0%\n",
            "Epoch 21 step 18000 loss 0.9963 acc. 70.0%\n",
            "Epoch 22 step 19000 loss 1.3021 acc. 62.0%\n",
            "Epoch 23 step 20000 loss 1.2631 acc. 64.0%\n",
            "Epoch 24 step 21000 loss 1.1915 acc. 70.0%\n",
            "Epoch 25 step 22000 loss 1.2094 acc. 65.0%\n",
            "Epoch 27 step 23000 loss 1.1537 acc. 64.0%\n",
            "Epoch 28 step 24000 loss 1.1064 acc. 69.0%\n",
            "Epoch 29 step 25000 loss 0.7520 acc. 74.0%\n",
            "Epoch 30 step 26000 loss 1.1119 acc. 70.0%\n",
            "Epoch 31 step 27000 loss 0.9707 acc. 73.0%\n",
            "Epoch 32 step 28000 loss 0.8661 acc. 71.0%\n",
            "Epoch 34 step 29000 loss 0.8486 acc. 72.0%\n",
            "Epoch 35 step 30000 loss 0.8491 acc. 75.0%\n",
            "Epoch 36 step 31000 loss 0.8097 acc. 71.0%\n",
            "Epoch 37 step 32000 loss 0.9395 acc. 66.0%\n",
            "Epoch 38 step 33000 loss 0.7833 acc. 71.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 70396 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 56.0%\n"
          ]
        }
      ],
      "source": [
        "# 2 more hidden layers\n",
        "net_advanced = torch.nn.Sequential(\n",
        "    torch.nn.InstanceNorm2d(1),\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 256),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(256, 128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_advanced.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_advanced.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_advanced, train_loader, epochs=40, lr=1e-3, momentum=0.9, plotting=True)\n",
        "test(net_advanced, test_loader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qYlKXg4RYTyj"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of parameters: 1749155\n",
            "Epoch 1 step 1000 loss 2.6367 acc. 28.0%\n",
            "Epoch 2 step 2000 loss 2.0656 acc. 37.0%\n",
            "Epoch 3 step 3000 loss 1.8378 acc. 47.0%\n",
            "Epoch 4 step 4000 loss 1.5597 acc. 53.0%\n",
            "Epoch 5 step 5000 loss 1.6491 acc. 49.0%\n",
            "Epoch 7 step 6000 loss 1.3775 acc. 49.0%\n",
            "Epoch 8 step 7000 loss 1.1088 acc. 66.0%\n",
            "Epoch 9 step 8000 loss 1.0239 acc. 68.0%\n",
            "Epoch 10 step 9000 loss 0.9516 acc. 65.0%\n",
            "Epoch 11 step 10000 loss 1.0081 acc. 65.0%\n",
            "Epoch 12 step 11000 loss 0.8299 acc. 72.0%\n",
            "Epoch 14 step 12000 loss 0.5035 acc. 82.0%\n",
            "Epoch 15 step 13000 loss 0.5228 acc. 85.0%\n",
            "Epoch 16 step 14000 loss 0.6687 acc. 80.0%\n",
            "Epoch 17 step 15000 loss 0.6879 acc. 79.0%\n",
            "Epoch 18 step 16000 loss 0.4961 acc. 87.0%\n",
            "Epoch 20 step 17000 loss 0.3369 acc. 92.0%\n",
            "Epoch 21 step 18000 loss 0.3571 acc. 87.0%\n",
            "Epoch 22 step 19000 loss 0.5278 acc. 84.0%\n",
            "Epoch 23 step 20000 loss 0.4016 acc. 83.0%\n",
            "Epoch 24 step 21000 loss 0.2504 acc. 92.0%\n",
            "Epoch 25 step 22000 loss 0.3364 acc. 86.0%\n",
            "Epoch 27 step 23000 loss 0.2140 acc. 93.0%\n",
            "Epoch 28 step 24000 loss 0.2170 acc. 93.0%\n",
            "Epoch 29 step 25000 loss 0.2232 acc. 93.0%\n",
            "Epoch 30 step 26000 loss 0.2727 acc. 92.0%\n",
            "Epoch 31 step 27000 loss 0.2087 acc. 94.0%\n",
            "Epoch 32 step 28000 loss 0.3686 acc. 87.0%\n",
            "Epoch 34 step 29000 loss 0.1660 acc. 96.0%\n",
            "Epoch 35 step 30000 loss 0.1798 acc. 95.0%\n",
            "Epoch 36 step 31000 loss 0.1974 acc. 93.0%\n",
            "Epoch 37 step 32000 loss 0.2400 acc. 90.0%\n",
            "Epoch 38 step 33000 loss 0.2257 acc. 94.0%\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/tmp/ipykernel_625190/1057436542.py:33: MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed in 3.11. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap()`` or ``pyplot.get_cmap()`` instead.\n",
            "  set1 = cm.get_cmap('Set1')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 800x500 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 65768 characters]",
            "text/plain": [
              "<Figure size 800x500 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Accuracy: 57.0%\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 343908 characters]",
            "text/plain": [
              "<Figure size 3900x3900 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# 4 more hidden layers\n",
        "net_advanced = torch.nn.Sequential(\n",
        "    torch.nn.InstanceNorm2d(1),\n",
        "    torch.nn.Flatten(),  # [B, 1, n_mels, LENGTH]  ->  [B, n_mels * LENGTH]\n",
        "    torch.nn.Linear(LENGTH * n_mels, 1024),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(1024, 512),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(512, 256),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(256, 128),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(128, 64),\n",
        "    torch.nn.ReLU(),\n",
        "    torch.nn.Linear(64, len(commands))\n",
        ")\n",
        "net_advanced.to(DEVICE)\n",
        "\n",
        "ps = sum(p.numel() for p in net_advanced.parameters())\n",
        "print(f\"Number of parameters: {ps}\")\n",
        "\n",
        "train(net_advanced, train_loader, epochs=40, lr=1e-2, momentum=0.9, plotting=True)\n",
        "test(net_advanced, test_loader)\n",
        "plot_confusion_matrix(net_advanced, test_loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WdTimQP1YTEe"
      },
      "source": [
        "## Analysis\n",
        "\n",
        "Here the spectrograms of the top 10 misclassifications are plotted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 220,
      "metadata": {
        "id": "7jz5C6cRYaVU"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top 10 most confident misclassifications:\n",
            "\n",
            "Sample 1: Predicted=six, Real=eight, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 20604 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 2: Predicted=eight, Real=cat, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 25168 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 3: Predicted=eight, Real=cat, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 22132 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 4: Predicted=eight, Real=cat, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 26912 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 5: Predicted=eight, Real=tree, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 22692 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 6: Predicted=happy, Real=backward, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 25160 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 7: Predicted=visual, Real=backward, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 22236 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 8: Predicted=seven, Real=happy, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 23852 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 38288 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 9: Predicted=no, Real=go, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 20048 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Sample 10: Predicted=dog, Real=go, Confidence=1.000\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 22844 characters]",
            "text/plain": [
              "<Figure size 640x480 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "\n",
              "                <audio  controls=\"controls\" >\n",
              "                    <source src=\"data:audio/wav;base64,[encoded data, 42728 characters]\" type=\"audio/wav\" />\n",
              "                    Your browser does not support the audio element.\n",
              "                </audio>\n",
              "              "
            ],
            "text/plain": [
              "<IPython.lib.display.Audio object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def worst_errors(model, test_loader, test_set, top_k=5):\n",
        "    errors = []\n",
        "    idx_offset = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for batch_idx, (features, labels) in enumerate(test_loader):\n",
        "            features = features.to(DEVICE)\n",
        "            labels = labels.to(DEVICE)\n",
        "            logits = model(features)\n",
        "            probs = torch.nn.functional.softmax(logits, dim=1)      # probs to see confidence of the model\n",
        "            preds = torch.argmax(probs, dim=1)\n",
        "            for i in range(len(labels)):\n",
        "                if preds[i] != labels[i]:\n",
        "                    confidence = probs[i, preds[i]].item()\n",
        "                    # Save: (confidence, batch_sample_idx, pred, real, global_sample_idx)\n",
        "                    errors.append((confidence, features[i].cpu(), preds[i].cpu().item(), labels[i].cpu().item(), idx_offset + i))\n",
        "            idx_offset += len(labels)\n",
        "\n",
        "    errors.sort(reverse=True, key=lambda x: x[0])\n",
        "    print(f\"Top {top_k} most confident misclassifications:\")\n",
        "\n",
        "    for i in range(min(top_k, len(errors))):\n",
        "        conf, feature, pred, real, global_idx = errors[i]\n",
        "        print(f\"\\nSample {i+1}: Predicted={commands[pred]}, Real={commands[real]}, Confidence={conf:.3f}\")\n",
        "        # Visualize spectrogram\n",
        "        plt.figure()\n",
        "        plt.imshow(feature[0], cmap=\"hot\")\n",
        "        plt.title(f\"Pred: {commands[pred]}, Real: {commands[real]}, Conf: {conf:.2f}\")\n",
        "        plt.colorbar()\n",
        "        plt.show()\n",
        "        # Play audio\n",
        "        waveform, sample_rate, *_ = test_set[global_idx]\n",
        "        display(ipd.Audio(waveform[0].numpy(), rate=sample_rate))\n",
        "\n",
        "worst_errors(net_advanced, test_loader, test_set, top_k=10)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "lab",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.13.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
