{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NEyumzI06iip"
      },
      "source": [
        "# Movie reviews\n",
        "\n",
        "Text classification finds application in countless domains, from spam filters in email clients, to document classification in search engines.  An interesting case of text classification is that of *sentiment analysis*.\n",
        "In sentiment analysis we want to predict the emotional state of the writers of text messages.\n",
        "It is a classification problem, where the classes are the authors' emotions (angry, delighted, sad, happy, disgusted...).\n",
        "Sentiment analysis has a large range of applications of significant economical and social value.\n",
        "It can be used as a cost-effective way of measuring the opinion of people by processing their comments about products, books, tv shows, political parties, etc.\n",
        "\n",
        "## Lab activity\n",
        "\n",
        "In this lab movie reviews written by the users of the IMDB website www.imdb.com will be analyzed, trying to predict the polarity of the reviews distinguishing the positive comments from the negative ones.\n",
        "A **multinomial Naïve Bayes classifier** will be used and, in the end, performance will be compared with a Linear SVM. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGVGoLC07VEQ"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The data set used is the same that has been introduced in the paper:\n",
        "\"Learning Word Vectors for Sentiment Analysis,\" by Andrew L. Maas et al.\n",
        "It includes 50000 reviews equally divided among the two\n",
        "classes; it's already divided in a training set of 25000 reviews, a validation set of 12500 and a test set of 12500.  A subset of the training set is also available to allow faster experiments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Code\n",
        "All the code from now on is organized to be reusable for later experiments with different parameters. The validation and test set have been used to operate a choice on which strategy is the best (stopwords, stemming, both etc.).\n",
        "\n",
        "Imports:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "qZ77_NXl6XQ5"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn.functional as F\n",
        "import matplotlib.pyplot as plt\n",
        "import collections\n",
        "import os\n",
        "import re\n",
        "\n",
        "import sys\n",
        "sys.path.append(\"aclImdb\")\n",
        "import porter"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eVk_lrCS_n72"
      },
      "source": [
        "### Build a vocabulary\n",
        "\n",
        "Reads all the training data and use it to build a list of the $n$ most frequent words. For now the most frequent 1000 words are selected."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "7UrhbYA0_-NQ"
      },
      "outputs": [],
      "source": [
        "VOCABULARY_SIZE = 1000\n",
        "\n",
        "def read_document(filename, stopwords=None, stemming=False):\n",
        "    \"\"\"Returns the list of words in the file.\"\"\"\n",
        "    words = []\n",
        "\n",
        "    with open(filename, \"r\") as f:\n",
        "        text = f.read()\n",
        "        text = re.sub(r\"\\W\",\" \",text, flags=re.UNICODE) # remove punctuation\n",
        "        words = text.lower().split()\n",
        "\n",
        "    # Remove stopwords and perform stemming if specified\n",
        "    if stopwords is not None:\n",
        "        words = [word for word in words if word not in stopwords]\n",
        "    if stemming:\n",
        "        words = [porter.stem(word) for word in words]\n",
        "    return words\n",
        "\n",
        "def build_vocabulary(voc_size, stopwords=None, stemming=False, verbose=False):\n",
        "    \"\"\"Builds a vocabulary of the most frequent words in the training set.\"\"\"\n",
        "    words = collections.Counter()\n",
        "    for dir in (\"pos\", \"neg\"):\n",
        "        for f in os.listdir(\"aclImdb/train/\" + dir):\n",
        "            words.update(read_document(\"aclImdb/train/\" + dir + \"/\" + f, stopwords=stopwords, stemming=stemming))\n",
        "\n",
        "    if verbose:\n",
        "        print(f\"The 10 most common words in a collection of {len(words)} words:\")\n",
        "        # Print the 10 most frequent words.\n",
        "        for word, count in words.most_common(10):\n",
        "            print(f\"{word}\\t{count}\")\n",
        "        print(\"\\n--------------------------------------------------\\n\")\n",
        "    \n",
        "\n",
        "    # This keeps the top VOCABULARY_SIZE words and associate them to an index.\n",
        "    vocabulary = {}\n",
        "    inverse = []  # inverse mapping\n",
        "    index = 0\n",
        "\n",
        "\n",
        "    for word, count in words.most_common(voc_size):\n",
        "        vocabulary[word] = index\n",
        "        inverse.append(word)\n",
        "        index += 1\n",
        "\n",
        "    return vocabulary, inverse\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The 10 most common words in a collection of 74891 words:\n",
            "the\t336749\n",
            "and\t164140\n",
            "a\t163123\n",
            "of\t145864\n",
            "to\t135724\n",
            "is\t107332\n",
            "br\t101871\n",
            "it\t96467\n",
            "in\t93976\n",
            "i\t87690\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "vocabulary, inverse = build_vocabulary(VOCABULARY_SIZE, verbose=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HfAUAE9LCkDC"
      },
      "source": [
        "### Extract the features\n",
        "\n",
        "Features are extracted using the *Bag of Words* (BoW) representation. The same procedure has been applied to the train, validation and test data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Sl8aPjFZDCaa"
      },
      "outputs": [],
      "source": [
        "def make_bow(filename, vocabulary):\n",
        "    \"\"\"Reads a document and return its BoW representation.\"\"\"\n",
        "    bow = torch.zeros(len(vocabulary), dtype=torch.int16)\n",
        "    \n",
        "    words = read_document(filename)\n",
        "    for word in words:\n",
        "        if word in vocabulary:\n",
        "            bow[vocabulary[word]] += 1\n",
        "    return bow\n",
        "\n",
        "\n",
        "def load_set(dirname, vocabulary):\n",
        "    \"\"\"Reads the content of a directory and return all features and labels.\"\"\"\n",
        "    features = []\n",
        "    labels = []\n",
        "\n",
        "    for subdir in (\"pos\", \"neg\"):\n",
        "        label = (1 if subdir == \"pos\" else 0)\n",
        "        for f in os.listdir(dirname + \"/\" + subdir):\n",
        "            bow = make_bow(dirname + \"/\" + subdir + \"/\" + f, vocabulary)\n",
        "            features.append(bow)\n",
        "            labels.append(label)\n",
        "    X = torch.stack(features)\n",
        "    Y = torch.tensor(labels)\n",
        "    return X, Y\n",
        "\n",
        "def build_sets(vocabulary):\n",
        "    \"\"\"Builds the training, validation and test sets and returns them.\"\"\"\n",
        "    Xtrain, Ytrain = load_set(\"aclImdb/train\", vocabulary)\n",
        "    Xvalid, Yvalid = load_set(\"aclImdb/validation\", vocabulary)\n",
        "    Xtest, Ytest = load_set(\"aclImdb/test\", vocabulary)\n",
        "\n",
        "    return Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dV5kcKsfSy0n"
      },
      "source": [
        "### Train the Multinomial Naïve Bayes classifier\n",
        "\n",
        "A multinomial Naïve Bayes with Laplacian smoothing is trained using this formula: \n",
        "$$\n",
        "  \\hat{y} = 1 \\iff\n",
        "  w^{T} x + b > 0, \\; \\; w_j = \\log \\pi_{1,j} - \\log \\pi_{0,j} \\;\\;\n",
        "  b = \\log P(1)  - \\log P(0).\n",
        "$$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "EKrR9fiJWsHX"
      },
      "outputs": [],
      "source": [
        "def train_nb(X, Y):\n",
        "    \"\"\"Trains a binary NB classifier.\"\"\"\n",
        "    X1 = (X * Y.unsqueeze(1)).float()\n",
        "    numerator = X1.sum(0) + 1           # Laplace smoothing gives the +1\n",
        "    denominator = X1.sum() + X.shape[1] # Laplace smoothing gives the +n, here the shape is len(vocabulary)\n",
        "    pi_1 = numerator / denominator\n",
        "\n",
        "    # Same for label 0\n",
        "    X0 = (X * (1 - Y.unsqueeze(1))).float()\n",
        "    numerator = X0.sum(0) + 1\n",
        "    denominator = X0.sum() + X.shape[1]\n",
        "    pi_0 = numerator / denominator\n",
        "\n",
        "    # The prior probabilities, in this case, are just the frequencies of the 0s and 1s\n",
        "    P1 = Y.float().mean()\n",
        "    P0 = 1 - P1\n",
        "    w = torch.log(pi_1) - torch.log(pi_0)\n",
        "    b = torch.log(P1) - torch.log(P0)\n",
        "\n",
        "    return w, b\n",
        "\n",
        "\n",
        "# Inference function\n",
        "def inference_nb(X, w, b):\n",
        "    Yhat = (X.float() @ w + b > 0).long()\n",
        "    return Yhat\n",
        "\n",
        "def predict_and_print(X, Y, w, b, spec_to_print=\"\"):\n",
        "    \"\"\"Predicts the labels of the data and prints the accuracy.\"\"\"\n",
        "    predictions = inference_nb(X, w, b)\n",
        "    accuracy = (predictions == Y).float().mean()\n",
        "    print(f\"{spec_to_print} Accuracy: {accuracy * 100:.2f}%\")\n",
        "    return accuracy\n",
        "\n",
        "def predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b):\n",
        "    \"\"\"Predicts and prints the accuracies on the training, validation and test sets.\"\"\"\n",
        "    train_acc = predict_and_print(Xtrain, Ytrain, w, b, \"Training\")\n",
        "    valid_acc = predict_and_print(Xvalid, Yvalid, w, b, \"Validation\")\n",
        "    test_acc = predict_and_print(Xtest, Ytest, w, b, \"Test\")\n",
        "    print(\"\\n--------------------------------------------------\\n\")\n",
        "    return train_acc, valid_acc, test_acc\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Accuracy: 80.45%\n",
            "Validation Accuracy: 80.16%\n",
            "Test Accuracy: 80.17%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "(tensor(0.8045), tensor(0.8016), tensor(0.8017))"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {},
      "outputs": [],
      "source": [
        "def find_most_impactful_words(w, inverse):\n",
        "    \"\"\"Finds the most impactful words in the model.\"\"\"\n",
        "    top_vals, top_indexes = torch.topk(w.abs(), k=100)\n",
        "    top_words = [inverse[i] for i in top_indexes]\n",
        "    print(\"Top words (both pos and neg): \", top_words)\n",
        "    print()\n",
        "    \n",
        "    # single argsort to distinguish pos and neg with values\n",
        "    indices = w.argsort(0)\n",
        "    print(\"NEGATIVE:\")\n",
        "    for i in range(10):\n",
        "        print(inverse[indices[i]], \"\\t\", w[indices[i]].item())\n",
        "\n",
        "    print()\n",
        "    print(\"POSITIVE:\")\n",
        "    for i in range(10):\n",
        "        print(inverse[indices[-i-1]], \"\\t\", w[indices[-i-1]].item())\n",
        "    \n",
        "    print(\"\\n--------------------------------------------------\\n\")\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Top words (both pos and neg):  ['waste', 'worst', 'awful', 'poorly', 'lame', 'horrible', 'crap', 'badly', 'worse', 'terrible', 'superb', 'mess', 'stupid', 'dull', 'wonderful', 'avoid', 'fantastic', 'ridiculous', 'boring', 'excellent', 'dumb', 'amazing', 'annoying', 'bad', 'unless', 'fails', 'powerful', 'supposed', 'favorite', 'poor', 'perfect', 'brilliant', 'joke', 'cheap', 'perfectly', 'loved', 'highly', 'today', 'unique', 'oh', 'beauty', 'greatest', 'predictable', 'sorry', 'oscar', 'save', 'beautiful', 'bunch', 'gore', 'heart', 'masterpiece', '7', 'minutes', '8', 'season', 'great', 'enjoyed', 'nudity', 'decent', 'moving', 'ok', 'memorable', 'episodes', 'strong', 'attempt', 'weak', 'apparently', 'nothing', 'couldn', 'money', 'realistic', 'total', 'zombie', 'unfortunately', '9', '4', 'tale', 'script', 'okay', 'monster', 'seriously', '1', 'enjoyable', 'performances', 'simple', 'premise', 'relationship', 'supporting', 'emotional', 'write', 'brings', 'guess', 'society', 'silly', 'best', 'instead', '2', 'effort', 'atmosphere', 'romantic']\n",
            "\n",
            "NEGATIVE:\n",
            "waste \t -2.6059632301330566\n",
            "worst \t -2.2785186767578125\n",
            "awful \t -2.217154026031494\n",
            "poorly \t -2.202465057373047\n",
            "lame \t -1.9666109085083008\n",
            "horrible \t -1.8997225761413574\n",
            "crap \t -1.818216323852539\n",
            "badly \t -1.7393684387207031\n",
            "worse \t -1.728630542755127\n",
            "terrible \t -1.7209620475769043\n",
            "\n",
            "POSITIVE:\n",
            "superb \t 1.7150135040283203\n",
            "wonderful \t 1.5693764686584473\n",
            "fantastic \t 1.5120067596435547\n",
            "excellent \t 1.469454288482666\n",
            "amazing \t 1.3979763984680176\n",
            "powerful \t 1.3061189651489258\n",
            "favorite \t 1.2720398902893066\n",
            "perfect \t 1.2516531944274902\n",
            "brilliant \t 1.235018253326416\n",
            "perfectly \t 1.2033376693725586\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "find_most_impactful_words(w, inverse)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [],
      "source": [
        "import textwrap\n",
        "\n",
        "def print_top_errors(Xtest, Ytest, w, b, print_file_contents=False):\n",
        "    \"\"\"Prints the 10 worst errors in the test set (both positive and negative, logits are in absolute value).\n",
        "    If print_file_contents is True, the contents of the files are printed as well. (Warning: large output!)\"\"\"\n",
        "    test_files = []\n",
        "    for dir in (\"pos\", \"neg\"):\n",
        "        for f in os.listdir(\"aclImdb/test/\" + dir):\n",
        "            test_files.append(f)\n",
        "\n",
        "\n",
        "    scores = Xtest.float() @ w + b\n",
        "    predictions = (scores > 0).long()\n",
        "    # if both are neg or pos, the score is positive\n",
        "    # if one is neg and the other pos, the score is negative\n",
        "    scores = scores * (2 * Ytest - 1).float() \n",
        "    # so with argsort we get the most negative scores first (the worst errors)\n",
        "    indexes = scores.argsort(0)\n",
        "    top_indexes = indexes[:10]\n",
        "\n",
        "    top_files = [test_files[i] for i in top_indexes]\n",
        "    print(\"WORST 10 ERRORS:\\n\")\n",
        "    print(\"Filename\\tPred\\tReal\\tValue\")\n",
        "    for file, pred, real, val in zip(top_files, predictions[top_indexes], Ytest[top_indexes], scores[top_indexes]):\n",
        "        print(f\"{file}\\t{pred.item()}\\t{real.item()}\\t{val.item()}\")\n",
        "\n",
        "    if print_file_contents:\n",
        "        for filename, pred, real, val in list(zip(top_files, predictions[top_indexes], Ytest[top_indexes], scores[top_indexes]))[:10]:\n",
        "            label_str = \"pos\" if real.item() == 1 else \"neg\"\n",
        "\n",
        "            print(\"\\n\")\n",
        "            with open(f\"aclImdb/test/{label_str}/{filename}\", \"r\") as f:\n",
        "                text = f.read()\n",
        "                wrapped_text = textwrap.fill(text, width=80)\n",
        "                print(wrapped_text)\n",
        "                print(\"\\n\")\n",
        "                f.close()\n",
        "                print(\"Filename: \", filename)\n",
        "                print(\"Predicted: \", pred.item())\n",
        "                print(\"Real: \", real.item())\n",
        "                print(\"Value: \", val.item())\n",
        "                print(\"\\n\")\n",
        "                print(\"--------------------------------------------------\")\n",
        "                print(\"\\n\")\n",
        "    print(\"\\n--------------------------------------------------\\n\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "WORST 10 ERRORS:\n",
            "\n",
            "Filename\tPred\tReal\tValue\n",
            "10784_4.txt\t1\t0\t-52.19377517700195\n",
            "11090_4.txt\t1\t0\t-51.32107925415039\n",
            "1376_8.txt\t0\t1\t-39.6142578125\n",
            "9154_2.txt\t1\t0\t-37.12382507324219\n",
            "2828_10.txt\t0\t1\t-33.5584716796875\n",
            "1922_1.txt\t1\t0\t-33.47578430175781\n",
            "8072_4.txt\t1\t0\t-32.66378402709961\n",
            "5522_3.txt\t1\t0\t-32.60936737060547\n",
            "6014_4.txt\t1\t0\t-30.23963737487793\n",
            "9244_4.txt\t1\t0\t-29.836986541748047\n",
            "\n",
            "\n",
            "The bearings of western-style Feminism on the various subcultures of India have\n",
            "hitherto remained largely non-existent, the two entities belonging to alien\n",
            "realms and threatening (in the name of tradition) never to coincide. Art\n",
            "imitates life (or so the claim goes) and popular Hindi cinema is no exception,\n",
            "reflecting an underlying misogyny which, regrettably, forms the foundation of\n",
            "much of the collective Indian culture. But why? What is it about the female\n",
            "gender that has rendered it so hateful to the culture that women are routinely\n",
            "subject to the most unimaginable horrors, including rape, murder, infanticide,\n",
            "imposed illiteracy, infidelity, and the subjugation of spirit that goes under\n",
            "the name of 'dowry'? Rajkumar Santoshi's latest offering, \"Lajja\", asks the same\n",
            "plaintive question, linking the atrocities committed against women through three\n",
            "separate chapters/episodes which comprise the journey of shame endured by its\n",
            "protagonist, Vaidehi (Manisha Koirala).<br /><br />Direction on Santoshi's part\n",
            "is not up to par with the level the story demands. He fails to achieve the\n",
            "necessary sensitivity in depicting the saga of sadness and confronts the issue\n",
            "of misogyny from the side, instead of head-on. Santoshi has recently said that\n",
            "he did not make the film for an international film festival, but rather for the\n",
            "masses of his country. Regrettably, the tackiness shows, and the film too often\n",
            "delves into the action-blood-gore genre that Santoshi specializes in. The film\n",
            "suffers from its jerky, episodic pace and its ending is rather too contrived.<br\n",
            "/><br />The female cast is given much kinder and more rounded characterizations\n",
            "than their male counterparts. The protagonist is played sensitively by the\n",
            "luminescently beautiful Manisha Koirala who proves in Lajja that she is one of\n",
            "our time's more competent leading ladies, and given a proper role and set up,\n",
            "emerges with a truly commendable performance. One wonders how brilliantly she\n",
            "may have shone had the film been made by a director with the appropriate\n",
            "creative intention and appreciation of the issue at hand. Mahima Choudhary puts\n",
            "in a laudable performance and continues to show that she is an untapped talent.\n",
            "Cast as Janki, Madhuri Dixit performs with a never-before-seen fervor and\n",
            "felicity for what truly deserves the name of 'acting.' The role of a street\n",
            "smart performer who finds solace in alcohol and the promise of an unborn child\n",
            "stands as the greatest risk in her cannon of song-n-dance roles which have\n",
            "maintained her marquee status over the past decade. Which leaves the final and\n",
            "most disturbing performance in this would-be feminist saga, that of the\n",
            "ceaselessly talented Rekha. Lajja is Manisha Koirala's film, there can be no\n",
            "doubt about that, but it is Rekha who dominates the proceedings in a performance\n",
            "that digs into your bones and sends echoes of terror through the vestibules of\n",
            "your heart. Rekha dazzles as Ramdulari, foregoing vanity and complacency to\n",
            "deliver a performance that is so replete with authenticity and ingenuity that\n",
            "emotional nudity becomes the mantra of this portion of the film. Comparisons are\n",
            "indeed odious, especially when rendered opposite one of the world's great\n",
            "leading ladies, but in the gracious presence of this reigning screen legend the\n",
            "others fade in her shadow.<br /><br />\"Lajja\" has none of the sophistication of\n",
            "proto-feminist dramas like \"Zubeidaa\", \"Pinjar\", or even the Hell-Queen\n",
            "celebration \"Laadla\": it fulfills its feministic goals in two early moments:the\n",
            "loud tirade in which Mahima berates her in-laws for their abuse of her father\n",
            "who has committed no other crime than given birth to a girl. She erupts, leaving\n",
            "the wedding procession in shambles. Seeing her father devastated, she begins to\n",
            "weep, blaming herself for the chaotic destruction in front of her. She bemoans,\n",
            "\"Why did I say anything? I have ruined everything! It is all my fault!\" Her\n",
            "grandmother, witnessing silently the abuse she bore, comforts her by saying,\n",
            "\"Why are you crying? There is no reason for you to be crying. You are not at\n",
            "fault for anything. The fault is mine. The fault is of every woman who came\n",
            "before you, because if we had had the courage to say in our day what you have\n",
            "said today, there would have been no need for you to say anything today.\" In\n",
            "this scene the importance of the Feminist Legacy is laid plainly in sight\n",
            "through words.<br /><br />The other, more subtle moment comes very early in the\n",
            "film when Vaidehi (Manisha) has fled from her abusive husband to the refuge of\n",
            "her parents' home in India. To viewers of western societies, it may seem\n",
            "perfectly reasonable (indeed, natural) that any abused woman would seek the\n",
            "protective guardianship of her parents; this, however, is a societal taboo in\n",
            "many eastern cultures, India among them. Once a woman has been married, the\n",
            "identity she assumes is that of her husband and his personal assets (family,\n",
            "business, children, etc.) For her to turn her back on these responsibilities is\n",
            "a grave social sin, one which truly has no equivalent for the western woman. She\n",
            "is thereafter regarded as tainted and as 'damaged goods', one whose value has\n",
            "been nullified entirely by her own actions and her refusal to submit to the role\n",
            "she has been given. She is not so much an individual as she is an emblem of\n",
            "familial honor. Her father rebukes her for her actions, concerned that his\n",
            "familial honor will be tarnished irreparably by the daughter he had already\n",
            "transferred to another man. His primary concern is that of the impending\n",
            "marriage of Vaidehi's younger sister, a prospect made far less likely with a\n",
            "divorced elder daughter in the same household. He tells her in no uncertain that\n",
            "she must return to the man to whom she lawfully belongs, however violent and\n",
            "sadistic he may be. He levies against her the age old adage that, \"The honor of\n",
            "every home lies in the hands of its daughter.\" Quietly and pensively, she\n",
            "replies, \"Yes, the honor of every home lies in the hands of its daughter. But\n",
            "there is no honor for the daughter herself.\"\n",
            "\n",
            "\n",
            "Filename:  10784_4.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -52.19377517700195\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Julie Delpy stars in this horrific film about a sadistic relationship between a\n",
            "father and a daughter in France of the 14th Century. The film attempts to\n",
            "shatter the romantic chivalry image of the heroic medieval knight, by showing a\n",
            "rather dreary image of the period, defined by psychological dysfunction, and\n",
            "violence. <br /><br />The movie opens with a child, François, growing up in the\n",
            "shadow of the Hundred Years' War, told by his father to keep his mother safe and\n",
            "to wait for his return. François takes action when he discovers his mother with\n",
            "a lover in bed. François murders him in the name of defending his father's\n",
            "honour. Like father like son, François grows up, and leaves his family, also to\n",
            "go to the same war. This setting is somewhat of an explanation for the events to\n",
            "come, as on his way home, we already notice that something is wrong with\n",
            "François. The war has not done well with him, he has changed.<br /><br />The\n",
            "daughter, Béatrice de Cortemart (Delpy), awaits her beloved father, to return\n",
            "from captivity of the English. She is pure of heart and she was left to take\n",
            "care of the estate while her father was gone. In her father's absence, Béatrice\n",
            "needs to deal with financial difficulties, which strengthens Béatrice's hope\n",
            "that her father will return to save her. But, upon his return, she notices that\n",
            "he lost the will to enjoy life, and he tortures and humiliates everything around\n",
            "him, even his own daughter. From this points the film depicts various ways how\n",
            "François torments his family. Starting with humiliating his own son, and ending\n",
            "with the rape of his own daughter, Béatrice.<br /><br />Setting the film in the\n",
            "Middle Ages supposed to soften the blow, as the viewer may tell himself, that\n",
            "these kind of violent acts were held in difficult times. And indeed, many films\n",
            "on the topic of Incest, such as Tim Roth's \"The War Zone (1999)\" which are\n",
            "contemporary were more shocking because of that.<br /><br />Delpy appears in\n",
            "this film in several daring nude scenes. Indeed she appears to be angelic and\n",
            "beautiful.<br /><br />I was annoyed when I saw some animal torture scenes. I\n",
            "believe, and this is not confirmed, that some birds were killed for the making\n",
            "of this film, which really upsets me. The quality of a film drops when real\n",
            "violence is used towards animals. I would hope that this movie will be re-\n",
            "released without those cruelty scenes. Those scenes do not contribute much to\n",
            "the film storyline.<br /><br />Overall, the movie is too long. The script is\n",
            "problematic. We don't get to see François and Béatrice before the war, we don't\n",
            "really get the answer why is he changed to such extreme. I would have pass on\n",
            "this film, however, I have to mention a few scenes that made this film worth\n",
            "watching:<br /><br />* Scenes of a young child being able to murder in cold\n",
            "blood is truly shocking. I saw it first time on \"City of God (2002)\". Here,\n",
            "François, murders his mother's lover, while his father away at war. Excellent\n",
            "scene and very graphic. * The scenes from Béatrice being raped by her father\n",
            "till she finds out she is pregnant from him are truly shocking and interesting.\n",
            "The scene after the rape, where Delpy burns her cloths and cleans herself. She\n",
            "asks her brother to kick her in the stomach with hopes to have a miscarriage.<br\n",
            "/><br />* The brother humiliation scenes where the father dumps his son's head\n",
            "into the food - humiliating him then ranting about the war. Later, dressing his\n",
            "son with women's cloths.<br /><br />The film won the César (French Oscar) for\n",
            "Best Costume Design, I agree, the costumes here really make the film look\n",
            "authentic for the time period. The movie location is Château de Puivert, a real\n",
            "12th century castle and a historical monument, located in Aude, South-Central\n",
            "France. Beautiful castle and mountain view, really helps you set into the period\n",
            "of this film. The film also nominated for 3 more César awards, but they were all\n",
            "snatched to the widely successful French film \"Au revoir, les enfants\"\n",
            "(\"Goodbye, Children\", 1998).<br /><br />--- Released as \"Beatrice\" in New York\n",
            "City, March 1987. Only to be screened in France on November 2007. Watched it on\n",
            "YES3 on 3 May 2007, 17:45, at work.\n",
            "\n",
            "\n",
            "Filename:  11090_4.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -51.32107925415039\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "i was having a horrid day but this movie grabbed me, and i couldn't put it down\n",
            "until the end... and i had forgotten about my horrid day. and the ending... by\n",
            "the way... where is the sequel!!!<br /><br />the budget is obviously extremely\n",
            "low... but ... look what they did with it! it reminds me of a play... they are\n",
            "basically working with a tent, a 'escape pod', a few guns, uniforms, camping\n",
            "gear, and a 'scanner' thing. that is it for props. Maybe this is even a good\n",
            "thing, forcing the acting and writing to have to step up and take their rightful\n",
            "place in film, as the centers of the work, instead of as afterthoughts used to\n",
            "have an excuse to make CGI fights (starwars).<br /><br />The cgi is fine. It is\n",
            "not exactly 'seamless'... but imho it still works. why? because there isn't too\n",
            "much of it, and what there is, is not 'taking over' with an army of effects\n",
            "house people trying to cram everything they can into the shot. it prompts the\n",
            "imagination... it's some relatively simple stuff, with decent composition\n",
            "(especially the heavy freighter shot.. there is one long shot that must be at\n",
            "least ten seconds...that tracks the entire length of the ship... it must be a\n",
            "record for sci fi battle sequence film making in the past 10 years, to have an\n",
            "action sequence that lasts longer than 0.75 seconds), and some relation to the\n",
            "story. it might look old or not 'state of the art', but it doesn't look stupid\n",
            "and it doesn't take away from the story.<br /><br />The acting is good, except\n",
            "the characters die too fast to get to know them. The captain was great, but a\n",
            "few of his scenes could have used another take. I also got confused with his\n",
            "character losing his cool and stomping on a corpse, I like to think captains are\n",
            "calm cool and in control... what was going on in that scene? did the other crew\n",
            "worry about him losing it at that moment? did he feel himself losing control?\n",
            "<br /><br />Now, as for the plot.... mostly it is good... why? Because it\n",
            "doesn't try to explain itself. It just happens. It's called 'the planet', its a\n",
            "mystery, get it?? Nobody knows why there is a statue, and they don't find out\n",
            "either. The mysterious cult? The weird scientist with the tattoo? What do you\n",
            "expect to find out in less than 90 minutes? This isn't War and Peace. And, thank\n",
            "god, it's not star wars/trek either. No midichlorians, no 5 minutes of\n",
            "expository boring dialog that has no purpose in the story. The characters are\n",
            "stranded, and are only able to figure out a few basic things... it is not a star\n",
            "trek episode where they find out it's leonardo davinci or a child like space\n",
            "wanderer. It is mysterious, and i liked that. I don't know why, maybe I can\n",
            "identify with these guys more , since they don't know whats happening, and i\n",
            "don't either... they don't talk a lot of space gibberish or have magic boxes\n",
            "telling them what is happening. <br /><br />In fact, I would argue that one of\n",
            "the weakest moments is when the 'traitor' turns on the crew, and tries to\n",
            "'explain' the reason for the planet, the cult, etc. This coincidentally has some\n",
            "of the weakest dialog, imho, in the whole movie, and it interrupts the flow and\n",
            "some of the characters look unnatural in that scene. <br /><br />OK, sometimes I\n",
            "felt it was a little too mysterious, though. Like, why did the guy get fried\n",
            "through his eyes with lightning? That was odd. Just weird. The 'hamlet'\n",
            "ending... again I would have liked to have known some of these characters\n",
            "better. And would it have been so hard to have a 30 second rescue scene at the\n",
            "end? This is not a serial show, it was a film, and we like closure in films,\n",
            "even if they can have a sequel. Imagine Hamlet with no 'flights of angels sing\n",
            "thee to thy rest'<br /><br />Anyways. What can I say. This was well worth the\n",
            "dollar I payed at the 'red box' machine at the supermarket. It was also, imho, a\n",
            "better piece of storytelling than starwars parts 1 2 or 3. Like I said, it\n",
            "sucked me in, wanting to know what was happening, and I couldn't stop watching\n",
            "until the end.\n",
            "\n",
            "\n",
            "Filename:  1376_8.txt\n",
            "Predicted:  0\n",
            "Real:  1\n",
            "Value:  -39.6142578125\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "THE MEMORY KEEPER'S DAUGHTER in the form of a novel by Kim Edwards was a highly\n",
            "successful bestseller and probably was featured in more reading groups than any\n",
            "other novel during its circulation. So what happened when the novel became a\n",
            "made-for-television movie? Perhaps it is the below mediocre screenplay (oops!,\n",
            "teleplay!) by John Pielmeier that consistently galumphs along in an awkward\n",
            "pedestrian fashion removing all sense of credibility to the story. Perhaps it is\n",
            "the cut and paste direction by Mick Jackson that misses the pacing and character\n",
            "delineation. Perhaps it suffers from the cinematography of an uncredited source\n",
            "or the 'liquid tears' musical score by Daniel Licht. For whatever of these (or\n",
            "all of these) reasons, this novel-to-film survives because it does make a good\n",
            "case for educating the public about the capabilities of those born with Down\n",
            "Syndrome. And for that it is worthy of attention. <br /><br />Dr. David Henry\n",
            "(Dermot Mulroney), a successful orthopedic doctor, is married to the beautiful\n",
            "Norah (Gretchen Mol) and their lives are becoming changed by their pregnancy. On\n",
            "a stormy winter night in Kentucky Norah goes into labor and the Henry's rush to\n",
            "a nearby clinic where David delivers his wife (the doctor is caught in a\n",
            "snowstorm) with the assistance of his old friend, nurse Caroline Gill (Emily\n",
            "Watson). After the delivery of a perfect boy child (Paul) Norah continues to be\n",
            "in labor and (surprisingly...) delivers an unexpected (!) twin girl. David and\n",
            "Caroline immediately recognize that the little girl (Phoebe) is a 'mongoloid'\n",
            "(this is before the use of the term Down Syndrome) and David, having a history\n",
            "of losing a little sister because of a birth defect) decides to send Phoebe to\n",
            "an asylum for the mentally challenged: Caroline is to make the delivery and\n",
            "Norah is told the second twin died at birth. <br /><br />Caroline follows\n",
            "instructions, sees the conditions of the 'home' where Phoebe is to be deposited,\n",
            "shrinks in horror, and decides to keep the child. Aided by a friendly trucker,\n",
            "Caroline changes her solitary existence and mothers Phoebe, finding a new life\n",
            "in her trucker's Pittsburgh. Norah insists on a formal funeral for Phoebe - a\n",
            "fact that deeply disturbs David's psyche, and the Henry's life goes on with only\n",
            "the one child Paul, leaving submerged pains about the lack of Phoebe's presence.\n",
            "Norah gifts David with a camera ('peoples lives are like a camera, that's where\n",
            "they live - in a room captured by a moment') and David becomes obsessed with\n",
            "photography. Norah grieves, drinks, and loses David's attention, while David\n",
            "traces Phoebe's existence with Caroline - sending money and letters to\n",
            "Pittsburgh. Paul (Tyler Stentiford to Jamie Spilchuk) grows up, discovers his\n",
            "mother's infidelities and is angered about his father's lack of communication\n",
            "and understanding, and decides to fulfill his goal of becoming a musician, and\n",
            "off to Juilliard he goes. Meanwhile Phoebe (Krystal Hope Nausbaum) has matured\n",
            "into a very highly adapted young girl, and the manner in which the broken\n",
            "marriage of the Henrys happens and the healing atmosphere of Phoebe's and Paul's\n",
            "lives coupled with the courage that has supported Caroline Gill's struggle to\n",
            "gain acceptance in the world for those born with Down Syndrome forms the\n",
            "conclusion of the film. <br /><br />The cast of well-known actors tries hard,\n",
            "but only Emily Watson is able to resurrect a credible character from this\n",
            "squishy script. Jamie Spilchuk gives evidence of a young actor with much\n",
            "promise. Dermot Mulroney and Gretchen find it difficult to mold empathetic\n",
            "characters form the corny lines they are given to deliver. The film is a mess,\n",
            "but the message about acceptance of Down Syndrome children and adults is an\n",
            "important one. Grady Harp\n",
            "\n",
            "\n",
            "Filename:  9154_2.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -37.12382507324219\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "The film had NO help at all, promotion-wise: if there was an advertising promo\n",
            "on TV or radio, I didn't see/hear it. The only newspaper ad I saw was on it's\n",
            "opening weekend: a dingy, sludgy B & W head-shot photo of Andy as Val-Com,\n",
            "behind jail bars, with headline: \"WANTED! Runaway Robot!\" ( which was also the\n",
            "poster in front of the 3 movie theaters I saw it at --NOT the nice little color\n",
            "poster on this site, with headshots of all the cast, and cartoon of Crimebuster\n",
            "--which really wasn't THAT good--they OUGHT to have used an action scene from\n",
            "the film itself--didn't they have an onset photographer? A poster is supposed to\n",
            "HELP a prospective audience decide if they want to SEE the movie--there were SO\n",
            "many people who couldn't get into their sold-out choice, and wanted to know WHAT\n",
            "Heartbeeps was about--and that poster didn't help! That dingy pic, and the only\n",
            "other photos supplied to papers were so indistinguishable in B & W that they\n",
            "were worthless. ) There was NO trailer for the film: only a slide at one\n",
            "theater, consisting of the word \"Heartbeeps\" inside a heart-shape, with a\n",
            "Cupid's arrow through it, and one that was a totally black picture: just Andy\n",
            "and Bernadette's voices saying \"Val-Com! My pleasure center is malfunctioning!\"\n",
            "\"So is mine; do you think we ought to tell our owners?\" THAT is no help to\n",
            "people who hadn't been aware of the movie.<br /><br />During the filming, Andy\n",
            "told reporters that he couldn't eat, once his plastic lips were applied, so he\n",
            "would \"load up on breakfast, and fast\" during the day's shoot. I don't know WHAT\n",
            "Bernadette did: but at the time, I'd wondered why they didn't just sip protein\n",
            "drinks through long straws, or eat astronaut-style puréed food via tubes? <br\n",
            "/><br />Phil-Co, the baby robot, seemed to have been the pre-curser to Short\n",
            "Circuit's Johnny-Five, with the same eyes, similar face. I've been trying to\n",
            "find if they had the same designer, but no help. I have vintage magazine\n",
            "articles about the film, and the design team was immensely proud of their work,\n",
            "and were going for a special award for their innovative device to create\n",
            "stenchless \"smoke\" for Catskill's cigars. Just shortly thereafter, LucasFilm did\n",
            "NOT use that device, though they OUGHT to have, for Return of the Jedi's scenes\n",
            "with Jabba the Hut: a man created \"steam\" around Jabba, by blowing cigar smoke\n",
            "into a tube, joking that all he needed was a glass of brandy, and he'd be a\n",
            "happy man. I thought that LucasFilm's using of real tobacco products was\n",
            "insensitive to people who were upset by smoke. <br /><br />John Williams, who\n",
            "had then recently succeeded the late, great Arthur Fielder as the maestro of the\n",
            "Boston Pops ( which was THEN a ratings hit--but it never recovered from\n",
            "Fielder's death, and is now a shadow of it's former glory ), was using the show\n",
            "to promote films with which he supplied the music. He'd premiered \"The Empire\n",
            "Strikes Back\" score there; and you would think he'd have helped Heartbeeps\n",
            "along, by playing a few numbers there? The one thing that critics had liked of\n",
            "this film was Williams' score--yet it was NOT available for purchase! I saw one\n",
            "vinyl album, in 1982, with half Heartbeeps, half another film--but it\n",
            "disappeared. I only just tonight saw the CD listed on THIS site, and have\n",
            "ordered it. If I can ever get a scanner, and time to type out the articles, I'd\n",
            "like to create a Heartbeeps tribute site. I liked the movie, and don't care what\n",
            "dissenters say! <br /><br />The only trouble with the film, was, that near the\n",
            "end, it was messed up, logic-wise: the robots ran away from the factory to have\n",
            "the freedom to decide their own fate, make their own choices; yet, when the\n",
            "junkyard owners tell them that Phil needs to go TO the factory, to have a\n",
            "\"purpose\" programmed into him, they don't even question it; they just glance\n",
            "meaningfully at each other, and they go. Along the way, each of the adults lose\n",
            "battery power, and \"die.\" They aren't REALLY dead, as they are robots, and only\n",
            "need new batteries, yet it is treated as \"death,\" with little Phil crying over\n",
            "them, and rolling away. So, what was the POINT of this? Phil never gets back to\n",
            "the factory, and gets \"a purpose!\" AND of course, the junkyard owners COULD'VE\n",
            "driven them, or given them all battery recharges, with back-up batteries; but\n",
            "the real point was to have this poignant scene, where the robots all wore down,\n",
            "and Phil is left to cry. <br /><br />At the end, Val-Com is a golf instructor,\n",
            "and Aqua-Com is --I'm not sure what. Catskill is an ENTERTAINER--what ELSE is HE\n",
            "supposed to be? I'm not sure that they made it clear. The junkyard owners seem\n",
            "to be taking it easy, lying on chaise lounges, drinking lemonade from Phil,\n",
            "their \"bartender.\" Val's and Aqua's new \"daughter,\" Philsia--I think the name is\n",
            "--maybe it's Sylvania--doesn't seem to be much more than a table lamp. <br /><br\n",
            "/>There is missing footage, which is sad--from photos I surmise that the stuff\n",
            "missing includes a sweet scene, where Phil is having a Christmas, with Val\n",
            "gifting him with a car's steering wheel; Aqua is supplying a horn; Catskill has\n",
            "taken the firefighter helmet to give to Phil, as we saw; and they have Christmas\n",
            "trees. I don't know if any missing footage supplies better logic, or if the\n",
            "writers just couldn't think of a better crisis/resolution. The film was trimmed\n",
            "to 72-75 minutes, to pair it with other failing films. No other reason than\n",
            "that. For a DVD, I would LOVE to be in on creating, as I want to see interviews\n",
            "with the cast/crew and John Williams, and the Merv Griffin interview. The\n",
            "making-of footage; and reediting and restoring the missing footage to make it\n",
            "better.\n",
            "\n",
            "\n",
            "Filename:  2828_10.txt\n",
            "Predicted:  0\n",
            "Real:  1\n",
            "Value:  -33.5584716796875\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "***SPOILERS*** ***SPOILERS*** HERE ON EARTH / (2000) 1/2* (out of four)<br /><br\n",
            "/>Mark Piznarski's \"Here on Earth\" holds the record for a movie containing the\n",
            "most recycled material in 96 minutes. Literally every contrivance, cliché, and\n",
            "familiar plot element are somewhere in here; there is simply nothing unique,\n",
            "original, or fresh about it. God, what an agonizing motion picture to sit\n",
            "through; I wish I saw the film during its theatrical release last year so it\n",
            "could have earned on my year's worst list. This is the kind of movie where the\n",
            "story makes itself instantly obvious, and goes downhill from the opening\n",
            "credits, and worst of all, takes itself seriously. \"Here on Earth\" is clearly\n",
            "one of the most horrible, painful movies to come down the pike in some time.<br\n",
            "/><br />\"Here on Earth\" is a teen heartthrob film, so it must contain some of\n",
            "Hollywood's most prized young men who are attractive and inexperienced enough to\n",
            "accept a role in a movie as bad as this. Chris Klein and Josh Hartnett fit that\n",
            "vary description, and take the lead roles by storm. Klein plays Kelley, an\n",
            "arrogant and insolent student with a wealthy father (cliché number one). He is\n",
            "to graduate as the valedictorian and attend Ivy league college following in the\n",
            "footsteps of his father (Stuart Wilson). This sets up the \"I don't want your\n",
            "life,\" cliché in which the father tries to control his son's life, while the son\n",
            "resists rebelliously. Throw in Kelley's deceased mother who committed suicide a\n",
            "while back. When his father brings home another woman, he brutally questions his\n",
            "intentions (the \"no one can replace mom\" cliché counts as number two).<br /><br\n",
            "/>Josh Hartnett plays Jasper, a character on the opposite side of society to\n",
            "Kelley. His family owns a local diner. Enter his long-time love interest,\n",
            "Samantha Cavanaugh (Leelee Sobieski) a waitress at the diner who covers for her\n",
            "sister (who has no purpose whatsoever rather than controlling the following\n",
            "scene) when Jasper and Kelley act like childish morons by racing their cars and\n",
            "(oh no) crashing into the diner, causing it to erupt into flames. (Conflicts\n",
            "between the rich and poor will count as cliché number three.)<br /><br />The\n",
            "bungled car chase sets both Kelley and Jasper up for a contrived and plausible\n",
            "conflict. They get in trouble with the law, but because this movie feels the\n",
            "need to exist, the local judge orders them an alternative to serving time: they\n",
            "must work together to help rebuild the diner.<br /><br />The two boys work hard\n",
            "during the summer growing strong and getting a nice tan. Samantha's eye catches\n",
            "Kelley, and romantic sparks fly. Jasper is jealous, but wants what is best for\n",
            "his true love (cliché number four). Her parents (Annette O'Toole and Bruce\n",
            "Greenwood) disapprove of her little romantic triangle (cliché number five), but\n",
            "she continues two timing Jasper without a second thought. Her father also\n",
            "happens to be the local sheriff, how surprising (lets count that melodramatic\n",
            "nugget cliché number six).<br /><br />The contrived romantic feelings between\n",
            "Kelley and Samantha count as cliché number seven. But Samantha's relationship\n",
            "with Jasper is never defined, so how can there any romantic tension? If the film\n",
            "is going to induce involvement in Samantha's choice between the two young men,\n",
            "then we need to see both characters from both sides. The movie depicts Jasper as\n",
            "a distraction to her \"rightful love,\" Kelley. It's clear Jasper truly loves her,\n",
            "but it is also clear she does not love him back. This absolutely slaughters the\n",
            "romantic tension early in the story.<br /><br />Leelee Sobieski does no harm\n",
            "here; however, her charm and kind performance do not fit a two-timing character\n",
            "like Samantha. John Hartnett is also right for the role of Jasper, but the movie\n",
            "gives him nothing to do except bicker with Kelley. Chris Klein gets to make a\n",
            "hunk name for himself here; he spends much of the movie shirtless, sweaty and\n",
            "overworked. Unfortunately he does not show off his acting ability, maybe because\n",
            "he has very little. The supporting cast is much more talented. Bruce Greenwood\n",
            "supplies the best performance in the film, but does not have near enough screen\n",
            "time to save anything but a few brief moments. I also enjoyed the performance by\n",
            "Stuart Wilson, who perfectly fits the role of a rich, controlling father of high\n",
            "social status.<br /><br />Then we lean about Samantha's knee problem exactly one\n",
            "hour and ten minutes into the movie (another spoiler ahead). What is this, she\n",
            "has a serious incurable illness (yet another contrivance into the picture,\n",
            "approximately number eight). Her terminal disease brings the two competing young\n",
            "men together as friends, well, at least I think that is what the movie intended\n",
            "to show, that the loss of one loved by two nemeses can bring both together\n",
            "(cliché number nine).<br /><br />Klein rehearses his valedictorian speech to\n",
            "demonstrate his character is more than a shallow stereotype, but we have seen\n",
            "this so many times before I would prefer a rich character rather than a deeply\n",
            "sentimental who hides actual feelings (cliché number, um, was it ten)?. The\n",
            "conflicts between Kelley and Jasper are desperate and inane; a \"your mom\"\n",
            "comment triggers a fist fight while they rebuild the establishment. There is a\n",
            "retread from \"Armageddon,\" as Samantha and Kelley sprawl out in an open field,\n",
            "horny as hell, as he slowly moves his fingers around her body, naming areas\n",
            "after US cities (why not call that number eleven). The movie uses alcohol as a\n",
            "means to increase the romantic tension: an intoxicated Kelley makes a fool of\n",
            "himself after getting in a fight with Samantha's date, Jasper, but the following\n",
            "day he recites desires only to dance (cliché number, oh no, I am losing count).\n",
            "\n",
            "\n",
            "Filename:  1922_1.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -33.47578430175781\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Cuore Sacro combines glossy film effects with a story that leaves much to be\n",
            "desired. With a script that the screen-writers for \"Touched by an Angel\" might\n",
            "have passed up as being too impuissant, Ozpetek still keeps us interested at\n",
            "times. In fact, I wanted to focus on the positives but I found the last act so\n",
            "bafflingly bizarre and awful that I think the couple who jumped to their deaths\n",
            "in the very beginning might have been the fortunate ones.<br /><br />This movie\n",
            "is at heart (pun intended) a story built on a big twist-style ending. This kind\n",
            "of tenuous foundation can result in a tremendous success like Tornatore's Una\n",
            "Pura Formalità or god-awful garbage like the films of M. Night Shyamalan. Cuore\n",
            "Sacro falls somewhat closer to the latter. I found the cinematography in general\n",
            "to be above average. The tracking shots of Irene dutifully doing her quotidian\n",
            "laps in the pool were very impressive as was the atmosphere conjured by the\n",
            "interior of her mother's house. For me, the grotesque parody of Michelangelo's\n",
            "Pieta when Giancarlo comes in from the rain and Irene poses with him was a bit\n",
            "of a stretch. One big issue that I took exception to in this film was Ozpetek's\n",
            "method of simply turning the camera directly into the face of his protagonist\n",
            "and recording the emotions taking place. This worked to fantastic effect in\n",
            "Facing Windows, but when employed here it seems that Bubolova is no Mezzogiorno.\n",
            "In fact besides the ridiculous story, the main problem with this film is the\n",
            "milquetoast performance of it's main character. It made the final breakdown\n",
            "scene even more unconscionably bad. <br /><br />In this movie Ozpetek continues\n",
            "his crusade against our corporate-driven societies by urging us to be more\n",
            "spiritual (not necessarily religious) and more altruistic. And while I'm\n",
            "certainly one who is very sympathetic to this view, I felt as if the audience\n",
            "was being hit over the head with a blunt object. Could the characters have been\n",
            "anymore two-dimensional? I tended to find this movie very enervating and\n",
            "soulless. Was the \"evil\" aunt Eleonora anything more than a caricature? It goes\n",
            "for the people on the side of \"right\" too, like the \"good\" aunt Maria Clara and\n",
            "the elderly doorman Aurelio. And just in case we might have missed Ozpetek's\n",
            "point, he decided to clothe his opposing forces in their own liveries. <br /><br\n",
            "/>This brings me to an interesting point about the director's use of color. He\n",
            "clothes the opening couple who briefly take flight in all black, as well as\n",
            "Irene (when we first meet her and after her life-conversion), the evil aunt\n",
            "Eleonora, and of course the good but confused Padre Carras. Black is a color\n",
            "that suggests a definite course, the wearer's mind is set and emotionless. It is\n",
            "the color of choice for that indispensable item of modern day armor, the\n",
            "business suit. It is also the color of mourning, such as the funerary finery\n",
            "sported by the suicidal duo. Finally, black is the color of piety, such as the\n",
            "simple robes of priests and nuns that Irene emulates in the second half of the\n",
            "film. <br /><br />The other main color, and a very appropriate choice for a\n",
            "movie about the sacred heart, is red. It is a color that has an extreme inherent\n",
            "emotional component. The character who wears red is bold, emotional, receptive\n",
            "to new ideas, and indulgent. Red is a risky color in modern times; it challenges\n",
            "our perceptions of the wearer and at the same time makes the wearer vulnerable.\n",
            "Yet red carries an enormous weight of history and mysticism, as the earliest\n",
            "members of Cro-Magnon man buried their dead in red ochre and indeed the first\n",
            "man named in the Torah, Adam, is named after the Hebrew word for red. Red also\n",
            "has an anachronistic flavor, looking back on the past where red (and by\n",
            "association a less self-driven attitude towards life) was more accepted. So when\n",
            "we encounter the red-filled room (the mysterious frieze covered walls complete\n",
            "with a red accented menorah and a red painting of a Whirling Dervish!) of\n",
            "Irene's mother, \"good\" characters Maria Clara and Aurelio wearing resplendent\n",
            "outfits of red, and finally the painting of Irene's mother in a formal red gown\n",
            "we can see where Ozpetek's sympathies lie.<br /><br />A word or two about the\n",
            "soundtrack, I found the original musical themes to be excellently suited to the\n",
            "story. The quasi-baroque theme that signified Irene was great for it's monotony\n",
            "and feeling of restive malaise (the absolute best use of a constantly repeated\n",
            "baroque theme such a this would have to be in Kubrick's Barry Lyndon, with it's\n",
            "masterful repetitions of an 8-bar sarabande attributed to Handel). One\n",
            "absolutely inspired choice was a couple of seconds of an opera aria we hear as\n",
            "the power is flickering while Irene is chasing Benny through the house. It is of\n",
            "the famous aria \"Ebben? ... Ne andrò lontano\" from Catalani's opera \"La Wally\".\n",
            "The aria is sung by the lead soprano who is leaving home forever. As Irene's\n",
            "mother was a dramatic soprano, we can guess that this is a recording of her\n",
            "singing and that she is saying a poignant farewell to her daughter, as in the\n",
            "movie Irene is soon destined to never again see Benny alive. I just have one\n",
            "minor question of the soundtrack, why include the famous tango Yo Soy Maria? I\n",
            "love the song and personally could hear it all the time, but it didn't really\n",
            "fit here.\n",
            "\n",
            "\n",
            "Filename:  8072_4.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -32.66378402709961\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\"Happy Days\" was produced and broadcast from the mid-1970's to the early 1980's\n",
            "and seems to get more ridiculous with age. At the time of its broadcast, most\n",
            "viewers who grew up in the 1950's were in middle age with families, and the\n",
            "scenes at Mel's Diner probably brought an artificial nostalgia to them. The Fonz\n",
            "was of course the coolest of the cool (although the actor Henry Wrinkler to this\n",
            "day has never learned how to ride a motorcycle). Richie Cunningham was the all-\n",
            "American blond-haired kid who would probably be elected student body president.\n",
            "Potsie was Richie's best friend--the star of the show has to have a best friend,\n",
            "I guess. And Ralph Malph was the bumbling sidekick to the Fonz, if not the\n",
            "entire group. I loved it when the Fonz would beat up on poor Ralph Malph. And\n",
            "there was Mel, the middle-aged lug who ran Mel's Diner. And of course who could\n",
            "forget the appearance of Mork? Was this really the 1950's? Ironically, films\n",
            "produced during the 1950's, such as \"Rebel Without a Cause\" and \"The Wild One\"\n",
            "have gotten better with age and portray the period more honestly than this show\n",
            "which was produced 20 years after the period it portrays.<br /><br\n",
            "/>Unfortunately, the TV show \"Happy Days\" is not in the same league as \"Rebel\n",
            "Without a Cause\" or \"American Graffitti\" for that matter. \"Happy Days\" may have\n",
            "captured some aspects of the 1950's with its burger diner, juke boxes, cool\n",
            "cars, and tacky plaid shirts, but it is more a nostalgic idealism done strictly\n",
            "for laughs rather than an honest portrayal. \"American Graffitti\" had something\n",
            "to say about young Americans in the 1950's whereas \"Happy Days\" seemed more\n",
            "about what middle-aged people of the 1970's wished the 1950's had been like. The\n",
            "result was a kind of watered down fabrication that really has nothing to do with\n",
            "the 1950's. \"Happy Days\" is, at best, a comedy-fantasy with some of the\n",
            "artificial culture of the 1950's as its backdrop. As pointed out by another\n",
            "reviewer, the all-American kid Richie Cunningham would probably have been\n",
            "chastised for befriending the likes of a drop-out like Fonzie. And Mel would\n",
            "probably forbid Fonzie from entering his Diner.<br /><br />A quick history:\n",
            "\"Happy Days\" was originally a pilot called \"Love in the Happy Days\" that was\n",
            "rejected for broadcast. Comedy pilots that had themes concerning sex and romance\n",
            "that did not make it to pilot airing sometimes appeared on the infrequently\n",
            "broadcast show \"Love American Style\" which was often aired in place of baseball\n",
            "games that had rained out or other unexpected programming cancellations and/or\n",
            "alterations. In short, \"Love American Style\" was a throw-away show that\n",
            "contained all these one-episode comedy pilots that never made it to a slotted\n",
            "debut. \"Love in the Happy Days\" did appear as a \"Love American Style\" show\n",
            "sometime in the early 1970's, but at the time TV executives could not foresee\n",
            "how a show about 1950's young people would be popular, particularly during the\n",
            "hey-day of comedy shows centering around middle-aged people, such as The \"Mary\n",
            "Tyler Moore Show\" (and its subsequent spin-offs such \"Rhoda\"), \"The Bob Newhart\n",
            "Show\", and \"All in the Family\". (How things have changed since now most TV\n",
            "sitcoms are about young people and the industry avoids most shows about middle-\n",
            "aged people like the plague!) <br /><br />Subsequently, one of the young stars\n",
            "of \"Love in the Happy Days\", a child actor from \"The Andy Griffith Show\" named\n",
            "Ron Howard, got the chance to star in a film about young people taking place in\n",
            "1959 called \"American Graffitti\" directed by the relatively unknown George Lucas\n",
            "whose previous \"THX 1138\" had bombed miserably at the box office. Even when it\n",
            "was premiered to movie executives, again the studios could not see how a movie\n",
            "about young people in the 1950's could become popular because it didn't \"fit\"\n",
            "with what had been popular in the past, although they didn't realize that much\n",
            "of the movie-going audience had been young in the 1950's. As everyone knows, the\n",
            "movie was a huge hit, and studio executives recognized that they had completely\n",
            "misjudged their audience. Somewhere during the theatrical run of \"American\n",
            "Graffitti\", TV executives realized they had a comedy pilot in their vault that\n",
            "was a lot like \"American Graffitti\". They brought it back with the original\n",
            "cast, plus Henry Wrinkler as \"The Fonz\", re-titled it \"Happy Days\" and the rest\n",
            "is TV history as it became one of the most popular shows of the 1970's.<br /><br\n",
            "/>\"Happy Days\" now seems ridiculous. The characters are flat and cardboard,\n",
            "never being more or less than what they superficially are. The issues they deal\n",
            "with are trivial. And their reactions appear mindless and even silly. Nowadays,\n",
            "the character of the Fonz seems to be a caricature of, well, The Fonz. Was the\n",
            "idea to be a kind of parody of Marlon Brando's character in \"The Wild One\"?\n",
            "Looking on the show with fresh eyes, I feel the producers really missed out on a\n",
            "great opportunity to present the 1950's with depth and realism that still could\n",
            "be fun and entertaining. Instead the producers decided on cheap laughs for quick\n",
            "bucks. This is definitely a show that has not withstood the test of time.\n",
            "\"American Graffitti\" has many of the outward appearances of \"Happy Days\" but it\n",
            "had an edge. It had an honesty about the characters and their issues. \"Happy\n",
            "Days\" took the look of \"American Graffitti\" but failed to take its heart.\n",
            "\n",
            "\n",
            "Filename:  5522_3.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -32.60936737060547\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Had she not been married to the producer, Jennifer Jones would not have been the\n",
            "most obvious choice for the leading female role in this tragic tale of an affair\n",
            "between an American soldier and an English nurse, set against the backdrop of\n",
            "the First World War. Her British accent is not perfect, and in the fifties it\n",
            "was unusual for a big romantic lead to go to an actress in her late thirties,\n",
            "even one as attractive as Miss Jones, especially when she was several years\n",
            "older than her leading man.. There were a number of beautiful young British\n",
            "actresses in Hollywood around this time, such as Audrey Hepburn, Elizabeth\n",
            "Taylor, Jean Simmons and Joan Collins, any of whom might have been more\n",
            "convincing in the role, but Miss Jones had one important attribute they all\n",
            "lacked, namely a marriage certificate with David O. Selznick's name on it. In\n",
            "the event, the film turned out to be such a turkey that they were doubtless\n",
            "grateful not to have it on their CVs. <br /><br />The film tells, at great\n",
            "length, the story of the romance between Frederick, an American volunteer\n",
            "serving with the Italian Army as an ambulance driver and Catherine, a nurse with\n",
            "the British Red Cross. After the Italian defeat at the battle of Caporetto,\n",
            "Frederick is wrongly accused of being a German spy and sentenced to death. (The\n",
            "film paints a very harsh picture of Italian military justice; it would appear\n",
            "that Italian Courts-Martial had the power to pass the death sentence after a\n",
            "trial lasting all of thirty seconds without hearing any evidence and without\n",
            "allowing the defendant to be legally represented or to speak in his defence).\n",
            "Frederick manages to escape and to cross the border into neutral Switzerland,\n",
            "accompanied by the pregnant Catherine. <br /><br />Hemingway's novels have not\n",
            "always been a great success when filmed. Howard Hawks succeeded in making a good\n",
            "version of \"To Have and have Not\", a film that is considerably better than the\n",
            "book on which it is nominally based, but that is because he largely ignored\n",
            "Hemingway's plot and turned the film into a remake of \"Casablanca\", set in\n",
            "Martinique rather than French Morocco. Like the 1943 version of \"For Whom the\n",
            "Bell Tolls\", \"A Farewell to Arms\" is overlong and fatally slow moving. It is\n",
            "also miscast. Jennifer Jones never makes Catherine come to life. As for Rock\n",
            "Hudson, his assumed Christian name could be unfortunately appropriate. He could\n",
            "be as solid as a rock but also as impassive as one, and in this film his\n",
            "Frederick seems an impersonation of the Great Stone Face. Despite the passion\n",
            "and emotion inherent in Hemingway's plot, the emotional temperature is always\n",
            "far too cool. The picture has little going for it apart from some attractive\n",
            "picture-postcard views of Italian and Swiss scenery. It is hardly surprising\n",
            "that it was not a success and that its failure ended Selznick's career as a\n",
            "producer. 4/10<br /><br />A goof. Shortly before the battle of Caporetto, an\n",
            "Italian officer states that Russia had already concluded a separate peace with\n",
            "Germany. That battle started in October 1917, at a time when Kerensky's Russia\n",
            "was still fighting alongside the Allies. The Russian Revolution did not take\n",
            "place until November; it was only the \"October Revolution\" by the old Julian\n",
            "calendar. The new Bolshevik regime signed an armistice with Germany in December\n",
            "1917, but a separate peace was not signed until the Treaty of Brest-Litovsk in\n",
            "March 1918\n",
            "\n",
            "\n",
            "Filename:  6014_4.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -30.23963737487793\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Korine's established himself, by now, as a talented and impressive image-maker.\n",
            "The promotional posters for Mister Lonely all include the film's most impressive\n",
            "compositions (though there's one in particular I've yet to see in promo\n",
            "material: that of a blue-clad nun teasing a dog with a stick, surrounded by\n",
            "green forest with torrential rain pouring down). The opening images of this\n",
            "film, of Michael Jackson lookalike (Diego Luna) riding a small motorbike round a\n",
            "track, is strangely compelling and beautiful: Roy Orbison's \"Mister Lonely\"\n",
            "plays on the soundtrack, and the images unfold in slow-motion. There's also a\n",
            "funny and terrific sequence in which the same character mimes a dance, without\n",
            "music (though a radio sits like a silent dog next to him), in the middle of a\n",
            "Paris street; Korine splices in sound effects and jump-cuts that evoke both a\n",
            "feeling of futility and dogged liberation in the character's dance routine.<br\n",
            "/><br />The first instance of the segment dealing with the nuns is also\n",
            "strangely poignant; Father Umbrillo (Werner Herzog) is an autocratic priest\n",
            "about to fly with some nuns over, and drop food into, impoverished areas nearby.\n",
            "In a scene that is both light-hearted and affecting, Herzog must deal with a\n",
            "stubbornly enthusiastic local who wishes to make the plane trip with them in\n",
            "order to see his wife in San Francisco. As the exchange develops, Herzog draws\n",
            "out of the man a confession: he has sinned, and his frequent infidelity is the\n",
            "cause of his wife having left him in the first place. This scene, short and\n",
            "sweet, gains particular weight after one learns its improvised origins: the\n",
            "sinner is played by a non-actor who was on set when Korine and co. were filming\n",
            "- and his adulterous ways had given him, in real life, a lasting, overwhelming\n",
            "guilt.<br /><br />Henceforth, the film is hit-and-miss; a succession of\n",
            "intrinsically interesting moments that add to a frivolous, muddled narrative.\n",
            "Whereas Gummo and Julien Donkey-Boy maintain their aesthetic and emotional\n",
            "weight via coherent structural frameworks, Mister Lonely feels like a victim of\n",
            "editing room ruthlessness. A few scenes were cut from the film, which would have\n",
            "otherwise painted fuller pictures of certain characters, due to continuity\n",
            "errors in costume - a result, no doubt, due to the absence of a shooting script\n",
            "and Korine's tendency for improvisation. One deleted scene in particular - in\n",
            "which 'Charlie Chaplin' (Denis Lavant) and 'Madonna' (Melita Morgan) have sex -\n",
            "would have added much more emotional conflict to a scene later on in the film (I\n",
            "won't spoil it, but it's there to deflate any feeling of warmth or celebration,\n",
            "and, as it is, only half-succeeds).<br /><br />The two strands of the narrative,\n",
            "unconnected literally, are best approached as two entirely different stories\n",
            "with the same allegorical meaning; one compliments the other and vice versa.\n",
            "(It's something to do with the conflict between one's ambitions and the reality\n",
            "of the current situation.) But there's not enough of the Herzog scenes to merit\n",
            "their place in the film, and so any connection between these two allegorically-\n",
            "connected threads is inevitably strained - and the inclusion is, in retrospect,\n",
            "tedious.<br /><br />This is an ambitious step forward from Julien Donkey-Boy\n",
            "that suffers mostly, at least in the lookalike segments, from having far too\n",
            "many characters for the film's running length, a flaw that would have been even\n",
            "worse had big star names played everyone (as was originally planned).<br /><br\n",
            "/>With many of the imagery's self-contained beauty, and moments of real, genuine\n",
            "connection with the soundtrack, this feels like it'd be much more suited to an\n",
            "art installation or photo exhibition. As an exploration of mimesis and the\n",
            "nature of impersonation, it'd lose none of its power - indeed, for me, it would\n",
            "perhaps be more impressive. The loneliness attached to iconic performativity\n",
            "(such as that encountered by both the icons themselves and those who aspire to\n",
            "be like them) is well-captured in images such as that wherein 'Marilyn Monroe'\n",
            "(a gorgeous Samantha Morton) seduces the camera with a Seven Year Itch pose in\n",
            "the middle of a forest, or when 'Sammy Davis, Jr.' (Jason Pennycooke) settles,\n",
            "post-dance rehearsal, with his back to the camera overlooking an incredible,\n",
            "tranquil lake.<br /><br />As it is, moments like these, and all those where the\n",
            "titles of randomly-chosen Michael Jackson songs crawl across the scene, are\n",
            "married to one another in a film narrative far less affecting than it should\n",
            "be.<br /><br />(For those who see it, I lost all faith during the egg-singing\n",
            "scene, late on. You'll know which scene I mean because it sticks out like a sore\n",
            "thumb, as some sort of gimmicky attempt at the new cinematic language for which\n",
            "Korine has previously been hailed.)\n",
            "\n",
            "\n",
            "Filename:  9244_4.txt\n",
            "Predicted:  1\n",
            "Real:  0\n",
            "Value:  -29.836986541748047\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print_top_errors(Xtest, Ytest, w, b, print_file_contents=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L6bXF_WUejyu"
      },
      "source": [
        "### Stopwords\n",
        "\n",
        "The vocabulary and the model are rebuilt to ignore very common words (those present in the `stopwords.txt` file, with the addition of \"br\"). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [],
      "source": [
        "f = open(\"aclImdb/stopwords.txt\")\n",
        "\n",
        "# here a set is used since it has O(1) lookup time instead of O(n) for a list\n",
        "# it's the best choice since the only operation performed on it is lookup\n",
        "stopwords = set(f.read().lower().split())\n",
        "stopwords.add(\"br\") # add \"br\" which is not in the stopwords list, but it's an html tag, not an actual review word\n",
        "f.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "4iqeT-QPfCJH"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Accuracy: 82.64%\n",
            "Validation Accuracy: 82.74%\n",
            "Test Accuracy: 82.26%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Top words (both pos and neg):  ['waste', 'pointless', 'worst', 'awful', 'poorly', 'lame', 'horrible', 'pathetic', 'wasted', 'crap', 'superb', 'badly', 'worse', 'terrible', 'mess', 'stewart', 'stupid', 'wonderful', 'dull', 'avoid', 'fantastic', 'excellent', 'ridiculous', 'boring', 'amazing', 'dumb', 'annoying', 'powerful', 'bad', 'unless', 'fails', 'favorite', 'perfect', 'zombies', 'brilliant', 'recommended', 'supposed', 'poor', 'perfectly', 'fake', 'joke', 'cheap', 'loved', 'highly', 'tony', 'today', 'awesome', 'unique', 'beauty', 'trash', 'oh', 'greatest', 'portrayal', 'incredible', 'failed', 'sweet', 'bored', 'oscar', 'solid', 'beautiful', 'predictable', 'heart', 'sorry', 'masterpiece', 'brain', '7', 'save', 'charming', '8', 'bunch', 'gore', 'flat', 'plain', 'season', 'great', 'minutes', 'natural', 'enjoyed', 'moving', 'memorable', 'smith', 'episodes', 'strong', 'adventure', 'f', 'nudity', 'realistic', 'decent', 'ok', 'attempt', 'weak', '9', 'tale', 'barely', 'apparently', 'couldn', 'animated', 'money', 'enjoyable', 'performances']\n",
            "\n",
            "NEGATIVE:\n",
            "waste \t -2.5806312561035156\n",
            "pointless \t -2.401174545288086\n",
            "worst \t -2.2531862258911133\n",
            "awful \t -2.1918206214904785\n",
            "poorly \t -2.1771316528320312\n",
            "lame \t -1.9412789344787598\n",
            "horrible \t -1.8743896484375\n",
            "pathetic \t -1.8546943664550781\n",
            "wasted \t -1.7959375381469727\n",
            "crap \t -1.7928829193115234\n",
            "\n",
            "POSITIVE:\n",
            "superb \t 1.7403459548950195\n",
            "stewart \t 1.643991470336914\n",
            "wonderful \t 1.594709873199463\n",
            "fantastic \t 1.5373401641845703\n",
            "excellent \t 1.4947867393493652\n",
            "amazing \t 1.423309326171875\n",
            "powerful \t 1.3314518928527832\n",
            "favorite \t 1.2973732948303223\n",
            "perfect \t 1.2769861221313477\n",
            "brilliant \t 1.2603516578674316\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "WORST 10 ERRORS:\n",
            "\n",
            "Filename\tPred\tReal\tValue\n",
            "11090_4.txt\t1\t0\t-36.705604553222656\n",
            "10784_4.txt\t1\t0\t-36.49732971191406\n",
            "5522_3.txt\t1\t0\t-30.80152130126953\n",
            "9154_2.txt\t1\t0\t-28.479846954345703\n",
            "6014_4.txt\t1\t0\t-26.603750228881836\n",
            "1376_8.txt\t0\t1\t-25.032909393310547\n",
            "6170_4.txt\t1\t0\t-24.879852294921875\n",
            "1922_1.txt\t1\t0\t-23.978822708129883\n",
            "2158_10.txt\t0\t1\t-23.83002281188965\n",
            "2918_1.txt\t1\t0\t-23.014175415039062\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# rebuild the vocabulary and the sets using the stopwords\n",
        "# and train the model \n",
        "vocabulary, inverse = build_vocabulary(VOCABULARY_SIZE, stopwords=stopwords)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "find_most_impactful_words(w, inverse)\n",
        "print_top_errors(Xtest, Ytest, w, b, print_file_contents=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I_l3unC8fDQB"
      },
      "source": [
        "### Stemming\n",
        "\n",
        "The vocabulary and the model are rebuilt applying the stemming algorithm (i.e. no distinction among words with the same root)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "x7-iDMH5fk2Z"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Accuracy: 77.46%\n",
            "Validation Accuracy: 76.67%\n",
            "Test Accuracy: 76.28%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Top words (both pos and neg):  ['worst', 'lame', 'chang', 'crap', 'suck', 'superb', 'mess', 'stupid', 'wai', 'georg', 'dull', 'avoid', 'realist', 'dumb', 'bother', 'wast', 'bad', 'unless', 'cliché', 'poor', 'perfect', 'joke', 'cheap', 'brilliant', 'rare', 'sever', 'ar', 'jame', 'ag', 'alon', 'dai', 'oh', 'attract', 'greatest', 'warn', 'save', 'oscar', 'bore', 'bunch', 'gore', 'heart', '7', '8', 'treat', 'season', 'great', 'excel', 'ha', 'decent', 'ok', 'attempt', 'weak', 'strong', 'couldn', 'mani', 'total', 'disappoint', '4', 'script', '9', 'tale', 'monster', '1', 'write', 'relationship', 'guess', 'alien', 'throw', 'instead', '2', 'best', 'effort', 'magic', 'zombi', 'shoot', 'fill', 'none', 'stick', 'rent', 'match', 'tri', 'ani', 'strang', 'ye', 'inde', 'gener', 'paul', 'favorit', 'boi', 'japanes', 'believ', 'noth', 'futur', 'parti', 'pai', 'love', 'award', '3', 'mean', 'master']\n",
            "\n",
            "NEGATIVE:\n",
            "worst \t -2.2839865684509277\n",
            "lame \t -1.9720792770385742\n",
            "crap \t -1.8236842155456543\n",
            "suck \t -1.7228050231933594\n",
            "mess \t -1.6815080642700195\n",
            "stupid \t -1.6529641151428223\n",
            "dull \t -1.5788674354553223\n",
            "avoid \t -1.571220874786377\n",
            "dumb \t -1.4207172393798828\n",
            "bother \t -1.4129743576049805\n",
            "\n",
            "POSITIVE:\n",
            "chang \t 1.9155597686767578\n",
            "superb \t 1.7095451354980469\n",
            "wai \t 1.6408653259277344\n",
            "georg \t 1.6080751419067383\n",
            "realist \t 1.5390825271606445\n",
            "perfect \t 1.2461848258972168\n",
            "brilliant \t 1.229550838470459\n",
            "rare \t 1.1582450866699219\n",
            "alon \t 1.097249984741211\n",
            "ag \t 1.097249984741211\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "WORST 10 ERRORS:\n",
            "\n",
            "Filename\tPred\tReal\tValue\n",
            "11090_4.txt\t1\t0\t-37.87916564941406\n",
            "1376_8.txt\t0\t1\t-35.98567199707031\n",
            "8072_4.txt\t1\t0\t-33.887969970703125\n",
            "9154_2.txt\t1\t0\t-27.8626651763916\n",
            "10784_4.txt\t1\t0\t-27.74053192138672\n",
            "10050_4.txt\t1\t0\t-23.78815269470215\n",
            "6014_4.txt\t1\t0\t-22.613361358642578\n",
            "5522_3.txt\t1\t0\t-22.431751251220703\n",
            "12420_10.txt\t0\t1\t-21.851776123046875\n",
            "8358_7.txt\t0\t1\t-21.812108993530273\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# pass stemming=True to rebuild the vocabulary and the sets\n",
        "# and train the model\n",
        "vocabulary, inverse = build_vocabulary(VOCABULARY_SIZE, stemming=True)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "find_most_impactful_words(w, inverse)\n",
        "print_top_errors(Xtest, Ytest, w, b, print_file_contents=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training Accuracy: 79.51%\n",
            "Validation Accuracy: 79.20%\n",
            "Test Accuracy: 77.78%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Top words (both pos and neg):  ['kei', 'pointless', 'worst', 'chang', 'lame', 'crap', 'superb', 'delight', 'wai', 'suck', 'georg', 'mess', 'stupid', 'realist', 'cloth', 'dull', 'avoid', 'dumb', 'bother', 'wast', 'bad', 'unless', 'perfect', 'brilliant', 'cliché', 'poor', 'rare', 'embarrass', 'fake', 'joke', 'cheap', 'dai', 'ag', 'greatest', 'ann', 'trash', 'jame', 'sweet', 'oh', 'oscar', 'attract', 'complex', 'solid', 'heart', '7', 'warn', '8', 'treat', 'paint', 'brain', 'season', 'save', 'bore', 'great', 'excel', 'bunch', 'gore', 'flat', 'plain', 'tear', 'smith', 'strong', 'disappoint', '9', 'tale', 'decent', 'ok', 'attempt', 'relationship', 'rip', 'weak', 'christian', 'couldn', 'best', 'magic', 'total', 'match', 'paul', 'believ', 'pai', 'japanes', 'futur', 'boi', 'favorit', 'parti', 'love', 'ride', '4', 'award', 'script', 'master', 'monster', '1', 'father', 'era', 'write', 'differ', 'nation', 'guess', 'william']\n",
            "\n",
            "NEGATIVE:\n",
            "pointless \t -2.38234281539917\n",
            "worst \t -2.234354019165039\n",
            "lame \t -1.922447681427002\n",
            "crap \t -1.7740516662597656\n",
            "suck \t -1.6731719970703125\n",
            "mess \t -1.6318750381469727\n",
            "stupid \t -1.6033320426940918\n",
            "dull \t -1.5292344093322754\n",
            "avoid \t -1.5215892791748047\n",
            "dumb \t -1.3710851669311523\n",
            "\n",
            "POSITIVE:\n",
            "kei \t 3.5447778701782227\n",
            "chang \t 1.9651927947998047\n",
            "superb \t 1.7591772079467773\n",
            "delight \t 1.72723388671875\n",
            "wai \t 1.6904983520507812\n",
            "georg \t 1.6577081680297852\n",
            "realist \t 1.5887155532836914\n",
            "cloth \t 1.5680961608886719\n",
            "perfect \t 1.2958173751831055\n",
            "brilliant \t 1.2791833877563477\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "WORST 10 ERRORS:\n",
            "\n",
            "Filename\tPred\tReal\tValue\n",
            "11090_4.txt\t1\t0\t-27.897855758666992\n",
            "1376_8.txt\t0\t1\t-21.563520431518555\n",
            "5522_3.txt\t1\t0\t-20.99990463256836\n",
            "8072_4.txt\t1\t0\t-20.436988830566406\n",
            "9154_2.txt\t1\t0\t-18.567548751831055\n",
            "6014_4.txt\t1\t0\t-18.4289608001709\n",
            "8136_4.txt\t1\t0\t-17.181655883789062\n",
            "11454_7.txt\t0\t1\t-16.817068099975586\n",
            "7834_8.txt\t0\t1\t-16.400545120239258\n",
            "12408_4.txt\t1\t0\t-16.148168563842773\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# both stopwords and stemming\n",
        "# as before, pass stopwords and stemming to rebuild the vocabulary and the sets\n",
        "# and train the model\n",
        "vocabulary, inverse = build_vocabulary(VOCABULARY_SIZE, stopwords=stopwords, stemming=True)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "find_most_impactful_words(w, inverse)\n",
        "print_top_errors(Xtest, Ytest, w, b, print_file_contents=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Vocabulary size\n",
        "Here 7 different sizes have been tested to determine which is the ideal one. The *stopwords-only* variant has been used, being the best one of the tests above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Vocabulary size: 100\n",
            "Training Accuracy: 71.34%\n",
            "Validation Accuracy: 71.40%\n",
            "Test Accuracy: 71.17%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 300\n",
            "Training Accuracy: 79.26%\n",
            "Validation Accuracy: 79.33%\n",
            "Test Accuracy: 79.46%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 500\n",
            "Training Accuracy: 80.90%\n",
            "Validation Accuracy: 80.98%\n",
            "Test Accuracy: 81.05%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 1000\n",
            "Training Accuracy: 82.64%\n",
            "Validation Accuracy: 82.74%\n",
            "Test Accuracy: 82.26%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 3000\n",
            "Training Accuracy: 84.81%\n",
            "Validation Accuracy: 83.57%\n",
            "Test Accuracy: 83.24%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 5000\n",
            "Training Accuracy: 85.60%\n",
            "Validation Accuracy: 83.22%\n",
            "Test Accuracy: 83.06%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Vocabulary size: 10000\n",
            "Training Accuracy: 86.74%\n",
            "Validation Accuracy: 82.86%\n",
            "Test Accuracy: 82.84%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# to test vocabulary size, the top from above will be used, so the stopwords only variant\n",
        "voc_sizes = [100, 300, 500, 1000, 3000, 5000, 10000]\n",
        "accs_dict = {}\n",
        "for voc_size in voc_sizes:\n",
        "    vocabulary, inverse = build_vocabulary(voc_size, stopwords=stopwords, verbose=False)\n",
        "    Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "    w, b = train_nb(Xtrain, Ytrain)\n",
        "    \n",
        "    print(f\"Vocabulary size: {voc_size}\")\n",
        "    train_acc, valid_acc, test_acc = predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "    accs_dict[voc_size] = {\n",
        "        \"train\": train_acc,\n",
        "        \"valid\": valid_acc,\n",
        "        \"test\": test_acc\n",
        "    }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Most impactful words at 3000:\n",
            "Training Accuracy: 84.81%\n",
            "Validation Accuracy: 83.57%\n",
            "Test Accuracy: 83.24%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Top words (both pos and neg):  ['boll', 'seagal', 'mst3k', 'matthau', 'victoria', 'unfunny', 'waste', 'blah', 'pointless', 'atrocious', 'redeeming', 'worst', 'lousy', 'laughable', 'awful', 'poorly', 'perfection', 'remotely', 'wasting', 'captures', 'existent', 'wonderfully', 'powell', 'lame', 'sucks', 'insult', 'uninteresting', 'horrible', 'lincoln', 'pathetic', 'unconvincing', 'amateurish', 'refreshing', 'idiotic', 'breathtaking', 'bourne', 'stupidity', 'wasted', 'crap', 'flynn', 'delightful', 'andrews', 'tedious', 'beautifully', 'dreadful', 'badly', 'elvira', 'worse', 'terrible', 'suck', 'underrated', 'gripping', 'superb', 'mess', 'embarrassing', 'garbage', 'delight', 'welles', 'stupid', 'pile', 'sinatra', 'touching', 'unforgettable', 'extraordinary', 'stewart', 'ashamed', 'brilliantly', 'favorites', 'sullivan', 'dull', 'avoid', 'friendship', 'wonderful', 'wooden', 'inept', 'vampires', 'magnificent', 'finest', 'jackie', 'forgettable', 'crappy', 'freedom', 'ridiculous', 'fantastic', 'sidney', 'bat', 'terrific', 'excuse', 'whatsoever', '0', 'noir', 'rubbish', 'boring', 'outstanding', 'nancy', 'marie', 'marvelous', 'junk', 'turkey', 'excellent']\n",
            "\n",
            "NEGATIVE:\n",
            "boll \t -4.279933929443359\n",
            "seagal \t -3.4242687225341797\n",
            "mst3k \t -3.0271711349487305\n",
            "unfunny \t -2.638350486755371\n",
            "waste \t -2.613337993621826\n",
            "blah \t -2.504704475402832\n",
            "pointless \t -2.4338817596435547\n",
            "atrocious \t -2.3740615844726562\n",
            "redeeming \t -2.3364152908325195\n",
            "worst \t -2.285892963409424\n",
            "\n",
            "POSITIVE:\n",
            "matthau \t 2.8101425170898438\n",
            "victoria \t 2.6823081970214844\n",
            "perfection \t 2.1638784408569336\n",
            "captures \t 2.040546417236328\n",
            "wonderfully \t 2.0221052169799805\n",
            "powell \t 1.980381965637207\n",
            "lincoln \t 1.9046630859375\n",
            "refreshing \t 1.8574838638305664\n",
            "breathtaking \t 1.851670265197754\n",
            "bourne \t 1.8506231307983398\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Most impactful words at 10000:\n",
            "Training Accuracy: 86.74%\n",
            "Validation Accuracy: 82.86%\n",
            "Test Accuracy: 82.84%\n",
            "\n",
            "--------------------------------------------------\n",
            "\n",
            "Top words (both pos and neg):  ['edie', 'antwone', 'din', 'boll', 'gunga', 'goldsworthy', 'gypo', 'yokai', 'paulie', 'uwe', 'slater', 'visconti', 'flavia', 'gino', 'kells', 'blandings', 'tashan', 'hobgoblins', 'brashear', 'deathtrap', 'kareena', 'kornbluth', 'panahi', 'harilal', 'gram', 'sarne', 'ossessione', 'caruso', 'tsui', 'delia', 'sabu', 'ahmad', 'saif', 'dominick', 'ae', 'khouri', 'aweigh', 'mj', 'btk', 'darkman', 'mcintire', 'kriemhild', 'shaq', 'tremors', 'blackie', 'newcombe', 'daisies', 'zenia', 'kei', 'trelkovsky', 'thunderbirds', 'seagal', 'beowulf', 'jaffar', 'bathsheba', 'hilliard', 'offside', 'pazu', 'sheeta', 'krell', 'gundam', 'dahmer', 'iturbi', 'wayans', 'grendel', 'ajay', 'giovanna', 'feinstone', 'felix', 'luzhin', 'philo', 'unwatchable', 'mst3k', 'lindy', 'stinker', 'dreck', 'capote', 'awfulness', 'fido', 'steaming', 'excellently', 'clutter', 'gamera', 'anchors', 'welch', 'vance', 'hayworth', 'kolchak', 'deanna', 'deathstalker', 'korda', 'sammo', 'callahan', 'incoherent', 'laputa', 'sox', 'biko', 'aiello', 'alvin', 'polanski']\n",
            "\n",
            "NEGATIVE:\n",
            "boll \t -4.302716255187988\n",
            "uwe \t -3.9578752517700195\n",
            "slater \t -3.9380722045898438\n",
            "tashan \t -3.8546905517578125\n",
            "hobgoblins \t -3.8546905517578125\n",
            "kareena \t -3.763718605041504\n",
            "kornbluth \t -3.71492862701416\n",
            "gram \t -3.66363525390625\n",
            "sarne \t -3.66363525390625\n",
            "delia \t -3.636967658996582\n",
            "\n",
            "POSITIVE:\n",
            "edie \t 4.674430847167969\n",
            "antwone \t 4.462586402893066\n",
            "din \t 4.392790794372559\n",
            "gunga \t 4.178643226623535\n",
            "goldsworthy \t 4.163604736328125\n",
            "yokai \t 4.0848236083984375\n",
            "gypo \t 4.0848236083984375\n",
            "paulie \t 4.059926986694336\n",
            "visconti \t 3.9251937866210938\n",
            "flavia \t 3.9251937866210938\n",
            "\n",
            "--------------------------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "#most impactful words at 3000 and 10000\n",
        "print(\"Most impactful words at 3000:\")\n",
        "vocabulary, inverse = build_vocabulary(3000, stopwords=stopwords, verbose=False)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "find_most_impactful_words(w, inverse)\n",
        "print(\"\\n--------------------------------------------------\\n\")\n",
        "print(\"Most impactful words at 10000:\")\n",
        "vocabulary, inverse = build_vocabulary(10000, stopwords=stopwords, verbose=False)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n",
        "w, b = train_nb(Xtrain, Ytrain)\n",
        "predict_and_print_all(Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest, w, b)\n",
        "find_most_impactful_words(w, inverse)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 78960 characters]",
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import seaborn as sns\n",
        "\n",
        "train_accuracies = [accs_dict[voc_size][\"train\"] for voc_size in voc_sizes]\n",
        "test_accuracies = [accs_dict[voc_size][\"test\"] for voc_size in voc_sizes]\n",
        "\n",
        "sns.set_theme(style=\"whitegrid\")\n",
        "plt.figure(figsize=(10, 6))\n",
        "plt.plot(voc_sizes, train_accuracies, label=\"Train Accuracy\", marker='o', color='blue', linestyle='--')\n",
        "plt.plot(voc_sizes, test_accuracies, label=\"Test Accuracy\", marker='s', color='green', linestyle='-')\n",
        "plt.xlabel(\"Vocabulary Size\", fontsize=14)\n",
        "plt.ylabel(\"Accuracy\", fontsize=14)\n",
        "plt.title(\"Train and Test Accuracy vs Vocabulary Size\", fontsize=16, fontweight='bold')\n",
        "plt.legend(fontsize=12)\n",
        "plt.grid(True, linestyle='--', alpha=0.7)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## SVM\n",
        "An SVM has been build to compare with the Naïve Bayes. Feature have been scaled to 0 mean and unit variance. Also, a scheduler has been used to gradually decrease learning rate during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 72,
      "metadata": {},
      "outputs": [],
      "source": [
        "def standardize(X):\n",
        "    \"\"\"\n",
        "    Standardize the input tensor X (zero mean, unit variance).\n",
        "    \"\"\"\n",
        "    mean = X.mean(dim=0, keepdim=True)\n",
        "    std = X.std(dim=0, keepdim=True)\n",
        "    X_standardized = (X - mean) / std\n",
        "    return X_standardized, mean, std\n",
        "\n",
        "def hinge_loss(z, y):\n",
        "    return torch.clamp((1 - 2 * y).float() * z, min=0).mean()\n",
        "\n",
        "# Arrays to store the values of the loss and accuracy for plotting\n",
        "steps_list = []\n",
        "losses = []\n",
        "accuracies = []\n",
        "\n",
        "def svm_train(X, Y, lambda_, lr=0.001, steps=1000):\n",
        "    m, n = X.shape\n",
        "    w = torch.zeros(n, requires_grad=True)\n",
        "    b = torch.zeros(1, requires_grad=True)\n",
        "    optimizer = torch.optim.SGD([w, b], lr=lr, weight_decay=lambda_)\n",
        "    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)\n",
        "    for step in range(steps):\n",
        "        optimizer.zero_grad()\n",
        "        z = X.float() @ w + b\n",
        "        Yhat = (z > 0).long()\n",
        "        accuracy = (Yhat == Y).float().mean()\n",
        "        loss = hinge_loss(z, Y)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        scheduler.step()\n",
        "\n",
        "        if step % 10 == 0:\n",
        "            print(f\"Step {step}: loss = {loss.item():.7f}\")\n",
        "            print(step, loss.item(), accuracy.item())\n",
        "            steps_list.append(step)\n",
        "            losses.append(loss.item())\n",
        "            accuracies.append(accuracy.item())\n",
        "    return w, b\n",
        "\n",
        "def svm_inference(X, w, b):\n",
        "    z = X.float() @ w + b\n",
        "    labels = (z > 0).long()\n",
        "    return labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 74,
      "metadata": {},
      "outputs": [],
      "source": [
        "vocabulary, inverse = build_vocabulary(3000, stopwords=stopwords)\n",
        "Xtrain, Ytrain, Xvalid, Yvalid, Xtest, Ytest = build_sets(vocabulary)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 75,
      "metadata": {},
      "outputs": [],
      "source": [
        "Xtrain, mean, std = standardize(Xtrain.float())\n",
        "Xvalid = (Xvalid.float() - mean) / std\n",
        "Xtest = (Xtest.float() - mean) / std"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 59,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Step 0: loss = 0.0000000\n",
            "0 0.0 0.5\n",
            "Step 10: loss = 0.0053621\n",
            "10 0.005362140014767647 0.7648400068283081\n",
            "Step 20: loss = 0.0005583\n",
            "20 0.000558264902792871 0.9152399897575378\n",
            "Step 30: loss = 0.0011425\n",
            "30 0.0011425267439335585 0.8940399885177612\n",
            "Step 40: loss = 0.0013070\n",
            "40 0.001306998310610652 0.854200005531311\n",
            "Step 50: loss = 0.0006289\n",
            "50 0.0006288528675213456 0.917680025100708\n",
            "Step 60: loss = 0.0046833\n",
            "60 0.004683312494307756 0.7640799880027771\n",
            "Step 70: loss = 0.0005046\n",
            "70 0.0005045874277129769 0.920199990272522\n",
            "Step 80: loss = 0.0019520\n",
            "80 0.0019519609631970525 0.8558800220489502\n",
            "Step 90: loss = 0.0007493\n",
            "90 0.0007493184530176222 0.8926799893379211\n",
            "Step 100: loss = 0.0007260\n",
            "100 0.0007259558769874275 0.9147999882698059\n",
            "Step 110: loss = 0.0005986\n",
            "110 0.0005986189353279769 0.9205600023269653\n",
            "Step 120: loss = 0.0005468\n",
            "120 0.000546818773727864 0.921280026435852\n",
            "Step 130: loss = 0.0005064\n",
            "130 0.0005064327269792557 0.9228799939155579\n",
            "Step 140: loss = 0.0004733\n",
            "140 0.0004732506931759417 0.9229599833488464\n",
            "Step 150: loss = 0.0004451\n",
            "150 0.0004450804553925991 0.9227200150489807\n",
            "Step 160: loss = 0.0004207\n",
            "160 0.00042072031646966934 0.9223999977111816\n",
            "Step 170: loss = 0.0003998\n",
            "170 0.0003997874155174941 0.9214800000190735\n",
            "Step 180: loss = 0.0003818\n",
            "180 0.00038176513044163585 0.9206799864768982\n",
            "Step 190: loss = 0.0003650\n",
            "190 0.00036497428664006293 0.9206399917602539\n",
            "Step 200: loss = 0.0003494\n",
            "200 0.00034940411569550633 0.9208400249481201\n",
            "Step 210: loss = 0.0003467\n",
            "210 0.00034668450825847685 0.9225199818611145\n",
            "Step 220: loss = 0.0003449\n",
            "220 0.00034493592102080584 0.9222800135612488\n",
            "Step 230: loss = 0.0003434\n",
            "230 0.000343379273544997 0.9228000044822693\n",
            "Step 240: loss = 0.0003419\n",
            "240 0.00034187932033091784 0.9225999712944031\n",
            "Step 250: loss = 0.0003404\n",
            "250 0.00034042453626170754 0.921720027923584\n",
            "Step 260: loss = 0.0003390\n",
            "260 0.0003390055790077895 0.9216399788856506\n",
            "Step 270: loss = 0.0003376\n",
            "270 0.00033758312929421663 0.9211999773979187\n",
            "Step 280: loss = 0.0003362\n",
            "280 0.00033622331102378666 0.9217600226402283\n",
            "Step 290: loss = 0.0003349\n",
            "290 0.0003348567115608603 0.9215199947357178\n",
            "Step 300: loss = 0.0003335\n",
            "300 0.00033346874988637865 0.9224399924278259\n",
            "Step 310: loss = 0.0003333\n",
            "310 0.0003332540800329298 0.9222000241279602\n",
            "Step 320: loss = 0.0003331\n",
            "320 0.0003330968611408025 0.9221199750900269\n",
            "Step 330: loss = 0.0003330\n",
            "330 0.0003329515748191625 0.9206399917602539\n",
            "Step 340: loss = 0.0003328\n",
            "340 0.0003328112361487001 0.9214800000190735\n",
            "Step 350: loss = 0.0003327\n",
            "350 0.00033267043181695044 0.9215999841690063\n",
            "Step 360: loss = 0.0003325\n",
            "360 0.0003325385332573205 0.9210000038146973\n",
            "Step 370: loss = 0.0003324\n",
            "370 0.0003323994460515678 0.9210799932479858\n",
            "Step 380: loss = 0.0003323\n",
            "380 0.0003322662669233978 0.9201599955558777\n",
            "Step 390: loss = 0.0003321\n",
            "390 0.00033212851849384606 0.9215199947357178\n",
            "Step 400: loss = 0.0003320\n",
            "400 0.00033199001336470246 0.9213600158691406\n",
            "Step 410: loss = 0.0003320\n",
            "410 0.0003319702227599919 0.9216799736022949\n",
            "Step 420: loss = 0.0003320\n",
            "420 0.00033195476862601936 0.9219200015068054\n",
            "Step 430: loss = 0.0003319\n",
            "430 0.00033194044954143465 0.9215999841690063\n",
            "Step 440: loss = 0.0003319\n",
            "440 0.0003319265379104763 0.920799970626831\n",
            "Step 450: loss = 0.0003319\n",
            "450 0.0003319128300063312 0.9205600023269653\n",
            "Step 460: loss = 0.0003319\n",
            "460 0.00033189967507496476 0.920960009098053\n",
            "Step 470: loss = 0.0003319\n",
            "470 0.00033188567613251507 0.9214000105857849\n",
            "Step 480: loss = 0.0003319\n",
            "480 0.0003318725503049791 0.9203199744224548\n",
            "Step 490: loss = 0.0003319\n",
            "490 0.0003318589006084949 0.9206799864768982\n",
            "Step 500: loss = 0.0003318\n",
            "500 0.00033184498897753656 0.9210399985313416\n",
            "Step 510: loss = 0.0003318\n",
            "510 0.0003318429517094046 0.9219599962234497\n",
            "Step 520: loss = 0.0003318\n",
            "520 0.00033184143831022084 0.9222800135612488\n",
            "Step 530: loss = 0.0003318\n",
            "530 0.00033184007043018937 0.9222400188446045\n",
            "Step 540: loss = 0.0003318\n",
            "540 0.000331838644342497 0.9221199750900269\n",
            "Step 550: loss = 0.0003318\n",
            "550 0.00033183724735863507 0.9219200015068054\n",
            "Step 560: loss = 0.0003318\n",
            "560 0.00033183596679009497 0.9215199947357178\n",
            "Step 570: loss = 0.0003318\n",
            "570 0.00033183456980623305 0.9222800135612488\n",
            "Step 580: loss = 0.0003318\n",
            "580 0.0003318332019262016 0.921720027923584\n",
            "Step 590: loss = 0.0003318\n",
            "590 0.00033183186315000057 0.9215199947357178\n",
            "Step 600: loss = 0.0003318\n",
            "600 0.0003318304952699691 0.9223600029945374\n",
            "Step 610: loss = 0.0003318\n",
            "610 0.0003318303788546473 0.922760009765625\n",
            "Step 620: loss = 0.0003318\n",
            "620 0.0003318302915431559 0.9218800067901611\n",
            "Step 630: loss = 0.0003318\n",
            "630 0.0003318301751278341 0.9218400120735168\n",
            "Step 640: loss = 0.0003318\n",
            "640 0.0003318301460240036 0.9213200211524963\n",
            "Step 650: loss = 0.0003318\n",
            "650 0.00033183005871251225 0.9227200150489807\n",
            "Step 660: loss = 0.0003318\n",
            "660 0.00033183000050485134 0.9235600233078003\n",
            "Step 670: loss = 0.0003318\n",
            "670 0.00033182994229719043 0.9218400120735168\n",
            "Step 680: loss = 0.0003318\n",
            "680 0.0003318298258818686 0.9229199886322021\n",
            "Step 690: loss = 0.0003318\n",
            "690 0.00033182979677803814 0.921720027923584\n",
            "Step 700: loss = 0.0003318\n",
            "700 0.0003318296803627163 0.9215999841690063\n",
            "Step 710: loss = 0.0003318\n",
            "710 0.0003318296803627163 0.9215599894523621\n",
            "Step 720: loss = 0.0003318\n",
            "720 0.0003318297094665468 0.9218800067901611\n",
            "Step 730: loss = 0.0003318\n",
            "730 0.0003318296803627163 0.9217600226402283\n",
            "Step 740: loss = 0.0003318\n",
            "740 0.0003318296803627163 0.9219599962234497\n",
            "Step 750: loss = 0.0003318\n",
            "750 0.0003318296803627163 0.9219599962234497\n",
            "Step 760: loss = 0.0003318\n",
            "760 0.0003318296803627163 0.9215599894523621\n",
            "Step 770: loss = 0.0003318\n",
            "770 0.0003318296803627163 0.9220799803733826\n",
            "Step 780: loss = 0.0003318\n",
            "780 0.0003318296803627163 0.9221600294113159\n",
            "Step 790: loss = 0.0003318\n",
            "790 0.0003318296803627163 0.9221199750900269\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 0 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "image/png": "[encoded data, 57148 characters]",
            "text/plain": [
              "<Figure size 800x600 with 2 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "steps_list = []\n",
        "losses = []\n",
        "accuracies = []\n",
        "\n",
        "w, b = svm_train(Xtrain, Ytrain, lambda_=1., lr=1e-2, steps=800)\n",
        "plt.figure()\n",
        "fig, axs = plt.subplots(2, 1, figsize=(8, 6))\n",
        "\n",
        "# Plot Loss\n",
        "axs[0].plot(steps_list, losses, label=\"Loss\", color=\"blue\")\n",
        "axs[0].set_title(\"Loss over Iterations\")\n",
        "axs[0].set_xlabel(\"Iterations\")\n",
        "axs[0].set_ylabel(\"Loss\")\n",
        "axs[0].legend()\n",
        "\n",
        "# Plot Accuracy\n",
        "axs[1].plot(steps_list, accuracies, label=\"Accuracy\", color=\"green\")\n",
        "axs[1].set_title(\"Accuracy over Iterations\")\n",
        "axs[1].set_xlabel(\"Iterations\")\n",
        "axs[1].set_ylabel(\"Accuracy\")\n",
        "axs[1].legend()\n",
        "\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 76,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train accuracy: 92.20%\n",
            "Test accuracy: 84.78%\n"
          ]
        }
      ],
      "source": [
        "test_predictions = svm_inference(Xtest, w, b)\n",
        "train_predictions = svm_inference(Xtrain, w, b)\n",
        "train_accuracy = (train_predictions == Ytrain).float().mean()\n",
        "print(f\"Train accuracy: {train_accuracy.item() * 100:.2f}%\")\n",
        "test_accuracy = (test_predictions == Ytest).float().mean()\n",
        "print(f\"Test accuracy: {test_accuracy.item() * 100:.2f}%\")\n",
        "#find_most_impactful_words(w, inverse)\n",
        "#print_top_errors(Xtest, Ytest, w, b, print_file_contents=False)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "lab",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
