{ "cells": [ { "cell_type": "markdown", "id": "e5d91831", "metadata": {}, "source": [ "# Final exam notebook\n", "\n", "**Subject:** Machine Learning and Deep Learning (KAN-CDSCO2004U) \n", "\n", "This notebook follows a complete machine learning methodology:\n", "\n", "- dataset description\n", "- exploratory data analysis\n", "- preprocessing\n", "- three own models\n", "- one advanced pretrained baseline\n", "- overfitting and underfitting analysis\n", "- feature importance and interpretability\n", "- model complexity comparison\n", "- results, practical insights, and ethical discussion\n", "\n", "## Research Question\n", "\n", "Can relatively simple models classify facial emotions from 48x48 grayscale face images, and how does a custom CNN compare with traditional machine learning models and a pretrained deep learning baseline?\n", "\n", "## Models Used\n", "\n", "| Type | Model | Role |\n", "|---|---|---|\n", "| Own model 1 | Logistic Regression | Traditional ML comparison |\n", "| Own model 2 | Random Forest | Traditional ML + feature importance |\n", "| Own model 3 | Custom CNN | Main model designed manually |\n", "| Baseline | ResNet50V2 | Advanced pretrained comparison only |\n", "\n", "The pretrained model is **not** the main solution. It is used only as a strong baseline for comparison." ] }, { "cell_type": "code", "execution_count": null, "id": "64cf0521", "metadata": {}, "outputs": [], "source": [ "# python -m pip install kagglehub tensorflow scikit-learn matplotlib seaborn pandas numpy pillow -q" ] }, { "cell_type": "code", "execution_count": null, "id": "cf2aae0f", "metadata": {}, "outputs": [], "source": [ "import os\n", "import time\n", "import random\n", "import warnings\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "from PIL import Image\n", "\n", "from sklearn.model_selection import train_test_split, RandomizedSearchCV, StratifiedKFold\n", "from sklearn.preprocessing import StandardScaler, LabelEncoder\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.metrics import (\n", " accuracy_score,\n", " classification_report,\n", " confusion_matrix,\n", " f1_score\n", ")\n", "from sklearn.utils.class_weight import compute_class_weight\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "RANDOM_STATE = 42\n", "random.seed(RANDOM_STATE)\n", "np.random.seed(RANDOM_STATE)\n", "\n", "plt.style.use(\"default\")\n", "sns.set_theme(style=\"whitegrid\")\n" ] }, { "cell_type": "markdown", "id": "75600ce7", "metadata": {}, "source": [ "## 1. Dataset Loading\n", "\n", "The dataset is loaded directly from Kaggle using `kagglehub`:\n", "\n", "\n", "\n", "FER-2013 contains grayscale face images of size `48x48`. The task is to classify each image into one of seven emotion classes:\n", "\n", "`Angry`, `Disgust`, `Fear`, `Happy`, `Sad`, `Surprise`, `Neutral`.\n", "\n", "The original dataset contains approximately:\n", "\n", "- `28,709` training images,\n", "- `3,589` public test images.\n", "\n", "To keep the notebook runnable during development, the variable `MAX_IMAGES_PER_CLASS` limits how many images are loaded per class. For final training, we set it to `None` in order to get the best accuracy." ] }, { "cell_type": "code", "execution_count": null, "id": "c35f20f4", "metadata": {}, "outputs": [], "source": [ "# Download the FER-2013 dataset with kagglehub. The returned path can differ by machine,\n", "# so the next cell searches for the actual train/test folders instead of hard-coding them.\n", "try:\n", " import kagglehub\n", "except ImportError as exc:\n", " raise ImportError(\n", " \"kagglehub is not installed. Run: pip install kagglehub\"\n", " ) from exc\n", "\n", "DATASET_HANDLE = \"msambare/fer2013\"\n", "dataset_path = Path(kagglehub.dataset_download(DATASET_HANDLE))\n", "print(\"Dataset path:\", dataset_path)\n", "print(\"Files/folders:\", [p.name for p in dataset_path.iterdir()])\n" ] }, { "cell_type": "code", "execution_count": null, "id": "291fffa2", "metadata": {}, "outputs": [], "source": [ "def find_split_dirs(root: Path):\n", " \"\"\"Find train and test folders inside the Kaggle dataset directory.\"\"\"\n", " # Kaggle datasets sometimes contain one extra nested folder, so search recursively.\n", " candidates = list(root.rglob(\"train\"))\n", " train_dir = next((p for p in candidates if p.is_dir()), None)\n", " candidates = list(root.rglob(\"test\"))\n", " test_dir = next((p for p in candidates if p.is_dir()), None)\n", "\n", " if train_dir is None or test_dir is None:\n", " raise FileNotFoundError(\"Could not find train/test directories in the dataset.\")\n", " return train_dir, test_dir\n", "\n", "train_dir, test_dir = find_split_dirs(dataset_path)\n", "print(\"Train directory:\", train_dir)\n", "print(\"Test directory:\", test_dir)\n", "\n", "# Folder names are used as the human-readable emotion labels.\n", "class_names = sorted([p.name for p in train_dir.iterdir() if p.is_dir()])\n", "print(\"Classes:\", class_names)\n" ] }, { "cell_type": "markdown", "id": "0b10449e", "metadata": {}, "source": [ "## 2. Exploratory Data Analysis\n", "\n", "The first step is to understand the class distribution and inspect example images. This is important because emotion datasets are often imbalanced. Some emotions, such as `disgust`, usually have fewer examples than emotions such as `happy`." ] }, { "cell_type": "code", "execution_count": null, "id": "2fe36bf1", "metadata": {}, "outputs": [], "source": [ "def count_images_by_class(split_dir: Path):\n", " rows = []\n", " for class_dir in sorted([p for p in split_dir.iterdir() if p.is_dir()]):\n", " n_images = len(list(class_dir.glob(\"*.jpg\"))) + len(list(class_dir.glob(\"*.png\")))\n", " rows.append({\"class\": class_dir.name, \"count\": n_images})\n", " return pd.DataFrame(rows)\n", "\n", "train_counts = count_images_by_class(train_dir)\n", "test_counts = count_images_by_class(test_dir)\n", "counts_df = train_counts.merge(test_counts, on=\"class\", suffixes=(\"_train\", \"_test\"))\n", "counts_df[\"total\"] = counts_df[\"count_train\"] + counts_df[\"count_test\"]\n", "counts_df" ] }, { "cell_type": "code", "execution_count": null, "id": "4b77ab7e", "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(1, 2, figsize=(14, 4))\n", "sns.barplot(data=train_counts, x=\"class\", y=\"count\", ax=axes[0], palette=\"viridis\")\n", "axes[0].set_title(\"Training Class Distribution\")\n", "axes[0].tick_params(axis=\"x\", rotation=30)\n", "\n", "sns.barplot(data=test_counts, x=\"class\", y=\"count\", ax=axes[1], palette=\"mako\")\n", "axes[1].set_title(\"Test Class Distribution\")\n", "axes[1].tick_params(axis=\"x\", rotation=30)\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "ffb464d4", "metadata": {}, "outputs": [], "source": [ "def sample_image_paths(split_dir: Path, class_name: str, n=5):\n", " paths = list((split_dir / class_name).glob(\"*.jpg\")) + list((split_dir / class_name).glob(\"*.png\"))\n", " return random.sample(paths, min(n, len(paths)))\n", "\n", "fig, axes = plt.subplots(len(class_names), 5, figsize=(10, 12))\n", "for row, class_name in enumerate(class_names):\n", " paths = sample_image_paths(train_dir, class_name, n=5)\n", " for col in range(5):\n", " ax = axes[row, col]\n", " ax.axis(\"off\")\n", " if col < len(paths):\n", " img = Image.open(paths[col]).convert(\"L\")\n", " ax.imshow(img, cmap=\"gray\")\n", " if col == 0:\n", " ax.set_title(class_name, loc=\"left\", fontsize=10)\n", "plt.suptitle(\"Sample FER-2013 Images by Class\", y=1.01)\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "4ad970e4", "metadata": {}, "outputs": [], "source": [ "# Pixel intensity analysis on a small random sample\n", "sample_pixels = []\n", "for class_name in class_names:\n", " for path in sample_image_paths(train_dir, class_name, n=80):\n", " img = np.array(Image.open(path).convert(\"L\"), dtype=np.float32)\n", " sample_pixels.extend(img.flatten())\n", "\n", "plt.figure(figsize=(8, 4))\n", "sns.histplot(sample_pixels, bins=50, kde=True, color=\"#2f6f73\")\n", "plt.title(\"Pixel Intensity Distribution\")\n", "plt.xlabel(\"Pixel value\")\n", "plt.ylabel(\"Frequency\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "1c2a32c1", "metadata": {}, "source": [ "### EDA Observations\n", "\n", "The dataset consists of low-resolution grayscale face images. The faces are centered, which helps the models because important regions such as eyes, eyebrows, and mouth are usually in similar positions.\n", "\n", "Important observations to discuss after running the cells:\n", "\n", "- if one class has much fewer samples, the model may learn it poorly,\n", "- similar emotions such as `sad`, `fear`, and `neutral` may be confused,\n", "- the low resolution limits fine facial detail,\n", "- lighting and face angle can affect prediction quality." ] }, { "cell_type": "markdown", "id": "fe8a8a40", "metadata": {}, "source": [ "## 3. Preprocessing\n", "\n", "For traditional machine learning models, each image is flattened from `48x48` into `2304` pixel features.\n", "\n", "For deep learning models, images are kept in spatial format:\n", "\n", "- Custom CNN input: `(48, 48, 1)`\n", "- ResNet50V2 input: RGB resized images, because pretrained ImageNet models expect 3-channel images.\n", "\n", "The current final run uses the full FER-2013 training and test folders (`MAX_IMAGES_PER_CLASS = None`). During development, this variable can be set to a smaller number to make the notebook faster.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "af672fba", "metadata": {}, "outputs": [], "source": [ "IMG_SIZE = (48, 48)\n", "MAX_IMAGES_PER_CLASS = None # Use None to train on the full FER-2013 dataset for best accuracy\n", "TEST_MAX_IMAGES_PER_CLASS = None # Use the full test set for final evaluation\n", "\n", "\n", "def load_images_from_directory(split_dir: Path, max_images_per_class=None):\n", " \"\"\"Load one FER split as normalized grayscale arrays and string labels.\"\"\"\n", " X = []\n", " y = []\n", " paths_used = []\n", "\n", " for class_name in sorted([p.name for p in split_dir.iterdir() if p.is_dir()]):\n", " paths = list((split_dir / class_name).glob(\"*.jpg\")) + list((split_dir / class_name).glob(\"*.png\"))\n", " paths = sorted(paths)\n", "\n", " # Optional per-class sampling keeps class balance when creating a faster development run.\n", " if max_images_per_class is not None:\n", " rng = random.Random(RANDOM_STATE)\n", " paths = rng.sample(paths, min(max_images_per_class, len(paths)))\n", "\n", " for path in paths:\n", " # Convert to grayscale, resize defensively, and scale pixels from 0-255 to 0-1.\n", " img = Image.open(path).convert(\"L\").resize(IMG_SIZE)\n", " arr = np.array(img, dtype=np.float32) / 255.0\n", " X.append(arr)\n", " y.append(class_name)\n", " paths_used.append(path)\n", "\n", " X = np.array(X, dtype=np.float32)\n", " y = np.array(y)\n", " return X, y, paths_used\n", "\n", "X_train_all, y_train_all, train_paths_used = load_images_from_directory(train_dir, MAX_IMAGES_PER_CLASS)\n", "X_test, y_test, test_paths_used = load_images_from_directory(test_dir, TEST_MAX_IMAGES_PER_CLASS)\n", "\n", "print(\"Training images:\", X_train_all.shape)\n", "print(\"Test images:\", X_test.shape)\n", "print(\"Training labels:\", y_train_all.shape)\n", "print(\"Test labels:\", y_test.shape)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d2e619a2", "metadata": {}, "outputs": [], "source": [ "# Encode text labels once, then use the same encoder for validation and test labels.\n", "label_encoder = LabelEncoder()\n", "y_train_all_enc = label_encoder.fit_transform(y_train_all)\n", "y_test_enc = label_encoder.transform(y_test)\n", "encoded_class_names = list(label_encoder.classes_)\n", "\n", "# Validation comes only from the original training split; the Kaggle test split stays untouched.\n", "X_train, X_val, y_train, y_val = train_test_split(\n", " X_train_all,\n", " y_train_all_enc,\n", " test_size=0.2,\n", " random_state=RANDOM_STATE,\n", " stratify=y_train_all_enc\n", ")\n", "\n", "print(\"Train:\", X_train.shape, y_train.shape)\n", "print(\"Validation:\", X_val.shape, y_val.shape)\n", "print(\"Test:\", X_test.shape, y_test_enc.shape)\n", "print(\"Encoded classes:\", encoded_class_names)\n" ] }, { "cell_type": "markdown", "id": "5fd4cf9c", "metadata": {}, "source": [ "### Preprocessing Sanity Checks\n", "\n", "Before training, these checks verify that the image scaling, shapes, labels, and stratified split are consistent. This reduces the chance of silent preprocessing mistakes.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d9dad7f0", "metadata": {}, "outputs": [], "source": [ "def class_distribution_percent(y_encoded):\n", " \"\"\"Return class percentages for one encoded split.\"\"\"\n", " counts = np.bincount(y_encoded, minlength=len(encoded_class_names))\n", " return pd.Series(counts / counts.sum(), index=encoded_class_names)\n", "\n", "# These assertions catch common preprocessing mistakes before model training starts.\n", "assert X_train_all.ndim == 3 and X_train_all.shape[1:] == IMG_SIZE\n", "assert X_test.ndim == 3 and X_test.shape[1:] == IMG_SIZE\n", "assert X_train.min() >= 0.0 and X_train.max() <= 1.0\n", "assert X_val.min() >= 0.0 and X_val.max() <= 1.0\n", "assert X_test.min() >= 0.0 and X_test.max() <= 1.0\n", "assert set(encoded_class_names) == set(class_names)\n", "assert len(np.unique(y_train)) == len(encoded_class_names)\n", "assert len(np.unique(y_val)) == len(encoded_class_names)\n", "\n", "# Similar percentages confirm that stratification preserved the class balance.\n", "split_distribution = pd.DataFrame({\n", " \"train_%\": class_distribution_percent(y_train),\n", " \"validation_%\": class_distribution_percent(y_val),\n", " \"test_%\": class_distribution_percent(y_test_enc)\n", "}).round(3)\n", "\n", "print(\"Preprocessing checks passed.\")\n", "display(split_distribution)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "8007442e", "metadata": {}, "outputs": [], "source": [ "# Class weights help the neural network pay more attention to minority classes.\n", "class_weights_array = compute_class_weight(\n", " class_weight=\"balanced\",\n", " classes=np.unique(y_train),\n", " y=y_train\n", ")\n", "class_weight_dict = {i: weight for i, weight in enumerate(class_weights_array)}\n", "\n", "pd.DataFrame({\n", " \"class\": encoded_class_names,\n", " \"class_weight\": [class_weight_dict[i] for i in range(len(encoded_class_names))]\n", "})\n" ] }, { "cell_type": "code", "execution_count": null, "id": "37446b4d", "metadata": {}, "outputs": [], "source": [ "# Flattened versions for traditional ML models\n", "X_train_flat = X_train.reshape(len(X_train), -1)\n", "X_val_flat = X_val.reshape(len(X_val), -1)\n", "X_test_flat = X_test.reshape(len(X_test), -1)\n", "\n", "print(\"Flattened feature count:\", X_train_flat.shape[1])" ] }, { "cell_type": "markdown", "id": "c0a38e37", "metadata": {}, "source": [ "## 4. Helper Functions for Evaluation\n", "\n", "The same evaluation functions are used for all models so that comparison is fair.\n", "\n", "Metrics used:\n", "\n", "- accuracy,\n", "- macro F1-score,\n", "- classification report,\n", "- raw and normalized confusion matrices,\n", "- training time,\n", "- prediction time.\n", "\n", "Macro F1 is useful because it treats all classes equally, which matters when the dataset is imbalanced. Normalized confusion matrices and the confusion-pair table make it easier to see which emotions are most often mixed up.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "75e1342e", "metadata": {}, "outputs": [], "source": [ "results = []\n", "\n", "\n", "def plot_confusion_matrix(y_true, y_pred, title, normalize=False):\n", " \"\"\"Plot either raw counts or row-normalized error rates.\"\"\"\n", " cm = confusion_matrix(y_true, y_pred)\n", " fmt = \"d\"\n", " values = cm\n", "\n", " # Row normalization answers: of all true examples in this class, where did they go?\n", " if normalize:\n", " row_totals = cm.sum(axis=1, keepdims=True)\n", " values = np.divide(cm, row_totals, out=np.zeros_like(cm, dtype=float), where=row_totals != 0)\n", " fmt = \".2f\"\n", " title = f\"{title} (Normalized by True Class)\"\n", "\n", " plt.figure(figsize=(8, 6))\n", " sns.heatmap(\n", " values,\n", " annot=True,\n", " fmt=fmt,\n", " cmap=\"Blues\",\n", " xticklabels=encoded_class_names,\n", " yticklabels=encoded_class_names\n", " )\n", " plt.title(title)\n", " plt.xlabel(\"Predicted\")\n", " plt.ylabel(\"True\")\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "\n", "def summarize_confusion_errors(y_true, y_pred, top_n=8):\n", " \"\"\"Return the most common off-diagonal confusion-matrix entries.\"\"\"\n", " cm = confusion_matrix(y_true, y_pred)\n", " rows = []\n", " for true_idx, true_name in enumerate(encoded_class_names):\n", " true_total = cm[true_idx].sum()\n", " for pred_idx, pred_name in enumerate(encoded_class_names):\n", " if true_idx == pred_idx or cm[true_idx, pred_idx] == 0:\n", " continue\n", " rows.append({\n", " \"true_class\": true_name,\n", " \"predicted_class\": pred_name,\n", " \"count\": int(cm[true_idx, pred_idx]),\n", " \"share_of_true_class\": cm[true_idx, pred_idx] / true_total if true_total else 0.0\n", " })\n", " columns = [\"true_class\", \"predicted_class\", \"count\", \"share_of_true_class\"]\n", " if not rows:\n", " return pd.DataFrame(columns=columns)\n", " return pd.DataFrame(rows, columns=columns).sort_values([\"count\", \"share_of_true_class\"], ascending=False).head(top_n)\n", "\n", "\n", "def evaluate_sklearn_model(name, model, X_train_data, y_train_data, X_val_data, y_val_data, X_test_data, y_test_data, model_type=\"Own model\"):\n", " \"\"\"Train and evaluate a traditional ML model using the same metrics for every model.\"\"\"\n", " start = time.perf_counter()\n", " model.fit(X_train_data, y_train_data)\n", " train_time = time.perf_counter() - start\n", "\n", " start = time.perf_counter()\n", " test_pred = model.predict(X_test_data)\n", " pred_time = time.perf_counter() - start\n", "\n", " # Validation is reported separately from the final test set.\n", " val_pred = model.predict(X_val_data)\n", "\n", " val_acc = accuracy_score(y_val_data, val_pred)\n", " test_acc = accuracy_score(y_test_data, test_pred)\n", " test_f1 = f1_score(y_test_data, test_pred, average=\"macro\")\n", "\n", " results.append({\n", " \"model\": name,\n", " \"type\": model_type,\n", " \"parameters\": \"N/A\",\n", " \"train_time_sec\": train_time,\n", " \"prediction_time_sec\": pred_time,\n", " \"val_accuracy\": val_acc,\n", " \"test_accuracy\": test_acc,\n", " \"test_macro_f1\": test_f1\n", " })\n", "\n", " print(f\"{name}\")\n", " print(f\"Validation accuracy: {val_acc:.4f}\")\n", " print(f\"Test accuracy: {test_acc:.4f}\")\n", " print(f\"Test macro F1: {test_f1:.4f}\")\n", " print(\"\\nClassification report:\")\n", " print(classification_report(y_test_data, test_pred, target_names=encoded_class_names))\n", " plot_confusion_matrix(y_test_data, test_pred, f\"{name} Confusion Matrix\")\n", " plot_confusion_matrix(y_test_data, test_pred, f\"{name} Confusion Matrix\", normalize=True)\n", " display(summarize_confusion_errors(y_test_data, test_pred))\n", " return model, test_pred\n" ] }, { "cell_type": "markdown", "id": "3af4ab8d", "metadata": {}, "source": [ "## 4.1 Hyperparameter Tuning with RandomizedSearchCV\n", "\n", "**What is hyperparameter tuning?**\n", "\n", "Hyperparameters are model settings chosen before training, for example the number of trees in Random Forest or the regularization strength in Logistic Regression. Hyperparameter tuning tests several possible settings and selects the combination that performs best.\n", "\n", "**Why RandomizedSearchCV?**\n", "\n", "`RandomizedSearchCV` tries a limited random sample of parameter combinations and evaluates them using cross-validation. It is usually faster than `GridSearchCV`, which tests every possible combination.\n", "\n", "In this notebook, tuning is used for the traditional machine learning models:\n", "\n", "- Logistic Regression\n", "- Random Forest\n", "\n", "For the CNN, tuning is done manually because full cross-validation would require training many neural networks and would be too slow for this assignment.\n", "\n", "To keep runtime reasonable, the search uses a stratified subset of the training data and a small number of random trials.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c0e6406e", "metadata": {}, "outputs": [], "source": [ "TUNING_SAMPLE_SIZE = 3500\n", "CV_FOLDS = 3\n", "N_RANDOM_SEARCH_ITER = 8\n", "\n", "# Defaults keep the final model cells runnable even if the search cells are skipped.\n", "best_log_reg_params = {}\n", "best_rf_params = {}\n", "\n", "\n", "def make_tuning_subset(X, y, max_samples=TUNING_SAMPLE_SIZE):\n", " \"\"\"Create a stratified subset so hyperparameter tuning remains runnable.\"\"\"\n", " if len(y) <= max_samples:\n", " return X, y\n", " _, X_small, _, y_small = train_test_split(\n", " X,\n", " y,\n", " test_size=max_samples,\n", " random_state=RANDOM_STATE,\n", " stratify=y\n", " )\n", " return X_small, y_small\n", "\n", "X_tune_flat, y_tune = make_tuning_subset(X_train_flat, y_train)\n", "cv_strategy = StratifiedKFold(n_splits=CV_FOLDS, shuffle=True, random_state=RANDOM_STATE)\n", "\n", "print(\"Tuning subset shape:\", X_tune_flat.shape)\n", "print(\"RandomizedSearchCV trials per model:\", N_RANDOM_SEARCH_ITER)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3967030e", "metadata": {}, "outputs": [], "source": [ "# Logistic Regression is tuned inside a Pipeline so scaling is refit inside each CV fold.\n", "log_reg_search = RandomizedSearchCV(\n", " estimator=Pipeline([\n", " (\"scaler\", StandardScaler()),\n", " (\"classifier\", LogisticRegression(\n", " solver=\"saga\",\n", " class_weight=\"balanced\",\n", " n_jobs=-1,\n", " random_state=RANDOM_STATE\n", " ))\n", " ]),\n", " param_distributions={\n", " \"classifier__C\": [0.05, 0.1, 0.25, 0.5, 1.0, 2.0],\n", " \"classifier__penalty\": [\"l2\"],\n", " \"classifier__max_iter\": [250, 400, 600]\n", " },\n", " n_iter=N_RANDOM_SEARCH_ITER,\n", " scoring=\"f1_macro\",\n", " cv=cv_strategy,\n", " n_jobs=-1,\n", " random_state=RANDOM_STATE,\n", " verbose=1\n", ")\n", "\n", "log_reg_search.fit(X_tune_flat, y_tune)\n", "best_log_reg_params = log_reg_search.best_params_\n", "\n", "print(\"Best Logistic Regression CV macro F1:\", log_reg_search.best_score_)\n", "print(\"Best Logistic Regression parameters:\", best_log_reg_params)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3f235211", "metadata": {}, "outputs": [], "source": [ "# Random Forest does not need feature scaling, so it is tuned directly.\n", "rf_search = RandomizedSearchCV(\n", " estimator=RandomForestClassifier(\n", " class_weight=\"balanced_subsample\",\n", " n_jobs=-1,\n", " random_state=RANDOM_STATE\n", " ),\n", " param_distributions={\n", " \"n_estimators\": [120, 180, 240, 300],\n", " \"max_depth\": [12, 18, 24, 32, None],\n", " \"min_samples_leaf\": [1, 2, 4],\n", " \"min_samples_split\": [2, 5, 10],\n", " \"max_features\": [\"sqrt\", \"log2\"]\n", " },\n", " n_iter=N_RANDOM_SEARCH_ITER,\n", " scoring=\"f1_macro\",\n", " cv=cv_strategy,\n", " n_jobs=-1,\n", " random_state=RANDOM_STATE,\n", " verbose=1\n", ")\n", "\n", "rf_search.fit(X_tune_flat, y_tune)\n", "best_rf_params = rf_search.best_params_\n", "\n", "print(\"Best Random Forest CV macro F1:\", rf_search.best_score_)\n", "print(\"Best Random Forest parameters:\", best_rf_params)\n" ] }, { "cell_type": "markdown", "id": "c5f53372", "metadata": {}, "source": [ "### 4.2 Sanity Baseline: Dummy Classifier\n", "\n", "A dummy classifier is included as a minimum baseline. It does not learn image patterns; it only predicts according to a simple rule. Real models should clearly outperform this baseline.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9e8de484", "metadata": {}, "outputs": [], "source": [ "dummy_baseline = DummyClassifier(strategy=\"most_frequent\", random_state=RANDOM_STATE)\n", "\n", "dummy_baseline, dummy_pred = evaluate_sklearn_model(\n", " \"Dummy Baseline\",\n", " dummy_baseline,\n", " X_train_flat,\n", " y_train,\n", " X_val_flat,\n", " y_val,\n", " X_test_flat,\n", " y_test_enc,\n", " model_type=\"Sanity baseline\"\n", ")\n" ] }, { "cell_type": "markdown", "id": "2da382d5", "metadata": {}, "source": [ "## 5. Own Model 1: Logistic Regression\n", "\n", "Logistic Regression is a simple traditional machine learning model. It receives flattened pixels as input, so it does not directly understand image structure.\n", "\n", "### Why include it?\n", "\n", "It gives a simple comparison point. If the CNN performs better, it supports the idea that spatial structure is important for image classification." ] }, { "cell_type": "code", "execution_count": null, "id": "ea461688", "metadata": {}, "outputs": [], "source": [ "log_reg_params = {\n", " \"max_iter\": 300,\n", " \"solver\": \"saga\",\n", " \"C\": 0.5,\n", " \"class_weight\": \"balanced\",\n", " \"n_jobs\": -1,\n", " \"random_state\": RANDOM_STATE\n", "}\n", "\n", "# If RandomizedSearchCV was run, use its best parameters.\n", "for key, value in globals().get(\"best_log_reg_params\", {}).items():\n", " clean_key = key.replace(\"classifier__\", \"\")\n", " log_reg_params[clean_key] = value\n", "\n", "log_reg = Pipeline([\n", " (\"scaler\", StandardScaler()),\n", " (\"classifier\", LogisticRegression(**log_reg_params))\n", "])\n", "\n", "log_reg, log_reg_pred = evaluate_sklearn_model(\n", " \"Logistic Regression\",\n", " log_reg,\n", " X_train_flat,\n", " y_train,\n", " X_val_flat,\n", " y_val,\n", " X_test_flat,\n", " y_test_enc\n", ")\n" ] }, { "cell_type": "markdown", "id": "fb66b950", "metadata": {}, "source": [ "## 6. Own Model 2: Random Forest\n", "\n", "Random Forest is another traditional machine learning model. It combines many decision trees and can model non-linear relationships.\n", "\n", "### Why include it?\n", "\n", "It is useful for comparison and provides a simple form of feature importance. In this project, feature importance tells us which pixel positions were most useful for classification." ] }, { "cell_type": "code", "execution_count": null, "id": "87a7abfb", "metadata": {}, "outputs": [], "source": [ "rf_params = {\n", " \"n_estimators\": 180,\n", " \"max_depth\": 24,\n", " \"min_samples_leaf\": 2,\n", " \"class_weight\": \"balanced_subsample\",\n", " \"n_jobs\": -1,\n", " \"random_state\": RANDOM_STATE\n", "}\n", "\n", "# If RandomizedSearchCV was run, use its best parameters.\n", "rf_params.update(globals().get(\"best_rf_params\", {}))\n", "\n", "random_forest = RandomForestClassifier(**rf_params)\n", "\n", "random_forest, rf_pred = evaluate_sklearn_model(\n", " \"Random Forest\",\n", " random_forest,\n", " X_train_flat,\n", " y_train,\n", " X_val_flat,\n", " y_val,\n", " X_test_flat,\n", " y_test_enc\n", ")\n" ] }, { "cell_type": "code", "execution_count": null, "id": "649f1076", "metadata": {}, "outputs": [], "source": [ "# Random Forest pixel feature importance\n", "importances = random_forest.feature_importances_.reshape(IMG_SIZE)\n", "\n", "plt.figure(figsize=(5, 5))\n", "plt.imshow(importances, cmap=\"inferno\")\n", "plt.title(\"Random Forest Pixel Importance\")\n", "plt.axis(\"off\")\n", "plt.colorbar(fraction=0.046, pad=0.04)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "6985c8fd", "metadata": {}, "source": [ "### Feature Importance Interpretation\n", "\n", "The Random Forest feature importance map shows which pixel positions helped most when splitting decision trees. For facial emotion recognition, important areas should often appear around:\n", "\n", "- the mouth,\n", "- eyes,\n", "- eyebrows,\n", "- central facial regions.\n", "\n", "This is not perfect semantic understanding, but it gives a useful classical ML explanation." ] }, { "cell_type": "markdown", "id": "a8279e7c", "metadata": {}, "source": [ "## 7. Own Model 3: Custom CNN\n", "\n", "This is the main model. It is designed manually using Keras/TensorFlow.\n", "\n", "### Why CNN?\n", "\n", "A CNN keeps the spatial structure of the image. Instead of treating each pixel independently, convolutional filters learn local patterns such as edges, shadows, mouth shape, and eye regions.\n", "\n", "### Architecture Idea\n", "\n", "The final CNN uses residual convolution blocks rather than a plain sequential stack. Each block uses convolution, batch normalization, ReLU activation, a residual shortcut, max pooling, and spatial dropout. This gives the model more capacity while L2 regularization, augmentation, early stopping, and learning-rate reduction help control overfitting.\n", "\n", "### Improvement strategy for this notebook\n", "\n", "- Train on the full FER-2013 dataset by setting `MAX_IMAGES_PER_CLASS = None`.\n", "- Use stronger data augmentation so the model sees more realistic variation.\n", "- Train for up to 70 epochs, but rely on early stopping to keep the best validation checkpoint.\n", "- Use residual convolution blocks for better feature learning than the earlier plain CNN.\n", "- Compare train, validation, and test metrics to check for overfitting.\n" ] }, { "cell_type": "markdown", "id": "0d463166", "metadata": {}, "source": [ "### Accuracy Improvement Strategy\n", "\n", "The notebook uses several methods from the machine learning and CNN exercises to improve generalization:\n", "\n", "- **normalization** scales pixel values to `[0, 1]`, which makes training more stable;\n", "- **data augmentation** creates slightly changed training images using flips, rotations, zoom, shifts, and contrast changes;\n", "- **class weights** reduce the effect of class imbalance by giving minority emotions more importance during training;\n", "- **batch normalization** stabilizes intermediate activations inside the CNN;\n", "- **dropout and L2 regularization** reduce overfitting;\n", "- **early stopping** keeps the best validation checkpoint;\n", "- **ReduceLROnPlateau** lowers the learning rate when progress slows down.\n", "\n", "In the saved output, these choices make the custom CNN the strongest model in the notebook, with about `59.5%` test accuracy and `0.552` macro F1. The score is not perfect, but it is a clear improvement over the traditional ML models and the dummy baseline.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "be478994", "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "from tensorflow.keras import layers, models, callbacks\n", "\n", "try:\n", " tf.keras.utils.set_random_seed(RANDOM_STATE)\n", "except Exception:\n", " pass\n", "\n", "num_classes = len(encoded_class_names)\n", "\n", "X_train_cnn = X_train[..., np.newaxis]\n", "X_val_cnn = X_val[..., np.newaxis]\n", "X_test_cnn = X_test[..., np.newaxis]\n", "\n", "print(\"CNN input shape:\", X_train_cnn.shape)" ] }, { "cell_type": "code", "execution_count": null, "id": "35c9dfe6", "metadata": {}, "outputs": [], "source": [ "# Augmentation is active only during training and helps the CNN generalize to small pose/lighting changes.\n", "data_augmentation = tf.keras.Sequential([\n", " layers.RandomFlip(\"horizontal\"),\n", " layers.RandomRotation(0.08),\n", " layers.RandomZoom(0.10),\n", " layers.RandomTranslation(0.05, 0.05),\n", " layers.RandomContrast(0.10),\n", "], name=\"data_augmentation\")\n", "\n", "\n", "def residual_conv_block(x, filters, dropout_rate):\n", " \"\"\"Convolution block with a shortcut connection and spatial dropout.\"\"\"\n", " shortcut = x\n", "\n", " x = layers.Conv2D(\n", " filters, 3, padding=\"same\", use_bias=False,\n", " kernel_regularizer=tf.keras.regularizers.l2(1e-4)\n", " )(x)\n", " x = layers.BatchNormalization()(x)\n", " x = layers.Activation(\"relu\")(x)\n", "\n", " x = layers.Conv2D(\n", " filters, 3, padding=\"same\", use_bias=False,\n", " kernel_regularizer=tf.keras.regularizers.l2(1e-4)\n", " )(x)\n", " x = layers.BatchNormalization()(x)\n", "\n", " # Project the shortcut when the number of channels changes.\n", " if shortcut.shape[-1] != filters:\n", " shortcut = layers.Conv2D(filters, 1, padding=\"same\", use_bias=False)(shortcut)\n", " shortcut = layers.BatchNormalization()(shortcut)\n", "\n", " x = layers.Add()([x, shortcut])\n", " x = layers.Activation(\"relu\")(x)\n", " x = layers.MaxPooling2D()(x)\n", " x = layers.SpatialDropout2D(dropout_rate)(x)\n", " return x\n", "\n", "\n", "inputs = layers.Input(shape=(IMG_SIZE[0], IMG_SIZE[1], 1))\n", "x = data_augmentation(inputs)\n", "\n", "# Filters increase by block so later layers can represent more abstract facial patterns.\n", "for filters, dropout_rate in [(32, 0.10), (64, 0.15), (128, 0.20), (256, 0.25)]:\n", " x = residual_conv_block(x, filters, dropout_rate)\n", "\n", "x = layers.GlobalAveragePooling2D()(x)\n", "x = layers.Dense(\n", " 256, activation=\"relu\",\n", " kernel_regularizer=tf.keras.regularizers.l2(1e-4)\n", ")(x)\n", "x = layers.BatchNormalization()(x)\n", "x = layers.Dropout(0.45)(x)\n", "outputs = layers.Dense(num_classes, activation=\"softmax\")(x)\n", "\n", "# Adam optimizer adapts the learning rate for each weight during training, which helps the CNN converge efficiently on this multi-class emotion classification task.\n", "custom_cnn = models.Model(inputs, outputs, name=\"custom_residual_fer_cnn\")\n", "custom_cnn.compile(\n", " optimizer=tf.keras.optimizers.Adam(learning_rate=7e-4), \n", " loss=\"sparse_categorical_crossentropy\",\n", " metrics=[\"accuracy\"]\n", ")\n", "custom_cnn.summary()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "ebf387a4", "metadata": {}, "outputs": [], "source": [ "cnn_callbacks = [\n", " callbacks.EarlyStopping(\n", " monitor=\"val_loss\",\n", " patience=8,\n", " restore_best_weights=True\n", " ),\n", " callbacks.ReduceLROnPlateau(\n", " monitor=\"val_loss\",\n", " factor=0.5,\n", " patience=3,\n", " min_lr=1e-6\n", " )\n", "]\n", "\n", "start = time.perf_counter()\n", "cnn_history = custom_cnn.fit(\n", " X_train_cnn,\n", " y_train,\n", " validation_data=(X_val_cnn, y_val),\n", " epochs=70,\n", " batch_size=64,\n", " callbacks=cnn_callbacks,\n", " class_weight=class_weight_dict,\n", " verbose=1\n", ")\n", "cnn_train_time = time.perf_counter() - start\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9ef5372b", "metadata": {}, "outputs": [], "source": [ "def plot_training_history(history, title):\n", " hist = pd.DataFrame(history.history)\n", " fig, axes = plt.subplots(1, 2, figsize=(13, 4))\n", "\n", " axes[0].plot(hist[\"accuracy\"], label=\"train\")\n", " axes[0].plot(hist[\"val_accuracy\"], label=\"validation\")\n", " axes[0].set_title(f\"{title} Accuracy\")\n", " axes[0].set_xlabel(\"Epoch\")\n", " axes[0].set_ylabel(\"Accuracy\")\n", " axes[0].legend()\n", "\n", " axes[1].plot(hist[\"loss\"], label=\"train\")\n", " axes[1].plot(hist[\"val_loss\"], label=\"validation\")\n", " axes[1].set_title(f\"{title} Loss\")\n", " axes[1].set_xlabel(\"Epoch\")\n", " axes[1].set_ylabel(\"Loss\")\n", " axes[1].legend()\n", "\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "plot_training_history(cnn_history, \"Custom CNN\")" ] }, { "cell_type": "code", "execution_count": null, "id": "dc6f5d81", "metadata": {}, "outputs": [], "source": [ "start = time.perf_counter()\n", "cnn_probs = custom_cnn.predict(X_test_cnn, verbose=0)\n", "cnn_pred_time = time.perf_counter() - start\n", "cnn_pred = np.argmax(cnn_probs, axis=1)\n", "\n", "cnn_val_loss, cnn_val_acc = custom_cnn.evaluate(X_val_cnn, y_val, verbose=0)\n", "cnn_test_loss, cnn_test_acc = custom_cnn.evaluate(X_test_cnn, y_test_enc, verbose=0)\n", "cnn_test_f1 = f1_score(y_test_enc, cnn_pred, average=\"macro\")\n", "\n", "results.append({\n", " \"model\": \"Custom CNN\",\n", " \"type\": \"Own model - main\",\n", " \"parameters\": custom_cnn.count_params(),\n", " \"train_time_sec\": cnn_train_time,\n", " \"prediction_time_sec\": cnn_pred_time,\n", " \"val_accuracy\": cnn_val_acc,\n", " \"test_accuracy\": cnn_test_acc,\n", " \"test_macro_f1\": cnn_test_f1\n", "})\n", "\n", "print(f\"Custom CNN validation accuracy: {cnn_val_acc:.4f}\")\n", "print(f\"Custom CNN test accuracy: {cnn_test_acc:.4f}\")\n", "print(f\"Custom CNN test macro F1: {cnn_test_f1:.4f}\")\n", "print(\"\\nClassification report:\")\n", "print(classification_report(y_test_enc, cnn_pred, target_names=encoded_class_names))\n", "plot_confusion_matrix(y_test_enc, cnn_pred, \"Custom CNN Confusion Matrix\")\n", "plot_confusion_matrix(y_test_enc, cnn_pred, \"Custom CNN Confusion Matrix\", normalize=True)\n", "display(summarize_confusion_errors(y_test_enc, cnn_pred))\n" ] }, { "cell_type": "markdown", "id": "da37632d", "metadata": {}, "source": [ "This CNN uses residual convolution blocks, L2 regularization, spatial dropout, batch normalization, data augmentation, early stopping, and learning-rate reduction. The goal is to improve validation accuracy while still controlling overfitting.\n" ] }, { "cell_type": "markdown", "id": "48395324", "metadata": {}, "source": [ "## 8. Advanced Baseline: ResNet50V2\n", "\n", "ResNet50V2 is a pretrained residual CNN architecture. It is used here as an advanced transfer-learning baseline, with a two-stage strategy that first learns a new classification head and then fine-tunes only a small number of final residual layers.\n", "\n", "### Why this is the baseline\n", "\n", "The pretrained model contains visual features learned from ImageNet, which can help when dataset size is limited. In this run, however, the custom CNN performs better than ResNet50V2. This is likely because FER-2013 images are very small, grayscale, and domain-specific, while ResNet50V2 was pretrained on large RGB natural images.\n", "\n", "Because ResNet50V2 expects RGB images, the grayscale FER images are converted from one channel to three channels and resized inside a `tf.data` pipeline. This avoids storing a very large resized RGB copy of the whole dataset in memory. During fine-tuning, BatchNormalization layers stay frozen, only the last 20 layers are unfrozen, and a very small learning rate is used to reduce overfitting.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1275c844", "metadata": {}, "outputs": [], "source": [ "USE_PRETRAINED_BASELINE = True\n", "PRETRAINED_SIZE = (128, 128)\n", "PRETRAINED_BATCH_SIZE = 64\n", "AUTOTUNE = tf.data.AUTOTUNE\n", "\n", "\n", "def make_pretrained_dataset(X_gray, y_encoded, batch_size=PRETRAINED_BATCH_SIZE, shuffle=False):\n", " \"\"\"Resize grayscale FER images to RGB batches for ResNet without storing a large RGB array.\"\"\"\n", " ds = tf.data.Dataset.from_tensor_slices((X_gray[..., np.newaxis], y_encoded))\n", " if shuffle:\n", " ds = ds.shuffle(buffer_size=min(len(y_encoded), 4096), seed=RANDOM_STATE, reshuffle_each_iteration=True)\n", "\n", " def preprocess_for_resnet(image, label):\n", " # Keep values in 0-1 here; the ResNet model cell applies preprocess_input after multiplying by 255.\n", " image = tf.cast(image, tf.float32)\n", " image = tf.image.grayscale_to_rgb(image)\n", " image = tf.image.resize(image, PRETRAINED_SIZE)\n", " return image, label\n", "\n", " return ds.map(preprocess_for_resnet, num_parallel_calls=AUTOTUNE).batch(batch_size).prefetch(AUTOTUNE)\n", "\n", "if USE_PRETRAINED_BASELINE:\n", " train_pretrained_ds = make_pretrained_dataset(X_train, y_train, shuffle=True)\n", " train_pretrained_eval_ds = make_pretrained_dataset(X_train, y_train, shuffle=False)\n", " val_pretrained_ds = make_pretrained_dataset(X_val, y_val, shuffle=False)\n", " test_pretrained_ds = make_pretrained_dataset(X_test, y_test_enc, shuffle=False)\n", "\n", " example_batch, _ = next(iter(train_pretrained_eval_ds))\n", " print(\"Pretrained model batch input:\", example_batch.shape)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c5d1eb2f", "metadata": {}, "outputs": [], "source": [ "if USE_PRETRAINED_BASELINE:\n", " base_model = tf.keras.applications.ResNet50V2(\n", " input_shape=(*PRETRAINED_SIZE, 3),\n", " include_top=False,\n", " weights=\"imagenet\"\n", " )\n", " base_model.trainable = False\n", "\n", " inputs = layers.Input(shape=(*PRETRAINED_SIZE, 3))\n", "\n", " # FER images are stored as 0-1 floats; ResNet50V2 preprocessing expects 0-255 style input.\n", " x = tf.keras.applications.resnet_v2.preprocess_input(inputs * 255.0)\n", " x = base_model(x, training=False)\n", " x = layers.GlobalAveragePooling2D()(x)\n", " x = layers.Dropout(0.45)(x)\n", " x = layers.Dense(\n", " 128, activation=\"relu\",\n", " kernel_regularizer=tf.keras.regularizers.l2(1e-4)\n", " )(x)\n", " x = layers.BatchNormalization()(x)\n", " x = layers.Dropout(0.40)(x)\n", " outputs = layers.Dense(num_classes, activation=\"softmax\")(x)\n", "\n", " pretrained_baseline = models.Model(inputs, outputs, name=\"resnet50v2_baseline\")\n", " pretrained_baseline.compile(\n", " optimizer=tf.keras.optimizers.Adam(learning_rate=7e-4),\n", " loss=\"sparse_categorical_crossentropy\",\n", " metrics=[\"accuracy\"]\n", " )\n", " pretrained_baseline.summary()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "6d654328", "metadata": {}, "outputs": [], "source": [ "if USE_PRETRAINED_BASELINE:\n", " start = time.perf_counter()\n", " pretrained_history = pretrained_baseline.fit(\n", " train_pretrained_ds,\n", " validation_data=val_pretrained_ds,\n", " epochs=12,\n", " callbacks=[\n", " callbacks.EarlyStopping(monitor=\"val_loss\", patience=4, restore_best_weights=True),\n", " callbacks.ReduceLROnPlateau(monitor=\"val_loss\", factor=0.5, patience=2, min_lr=1e-6)\n", " ],\n", " class_weight=class_weight_dict,\n", " verbose=1\n", " )\n", " pretrained_train_time = time.perf_counter() - start\n", " plot_training_history(pretrained_history, \"ResNet50V2 Baseline\")\n", "\n", " # Fine-tune conservatively: fewer ResNet layers, lower LR, frozen BatchNorm.\n", " base_model.trainable = True\n", " fine_tune_at = len(base_model.layers) - 20\n", " for layer in base_model.layers[:fine_tune_at]:\n", " layer.trainable = False\n", " for layer in base_model.layers[fine_tune_at:]:\n", " layer.trainable = True\n", " for layer in base_model.layers:\n", " if isinstance(layer, layers.BatchNormalization):\n", " layer.trainable = False\n", "\n", " pretrained_baseline.compile(\n", " optimizer=tf.keras.optimizers.Adam(learning_rate=3e-6),\n", " loss=\"sparse_categorical_crossentropy\",\n", " metrics=[\"accuracy\"]\n", " )\n", "\n", " start_fine = time.perf_counter()\n", " pretrained_fine_tune_history = pretrained_baseline.fit(\n", " train_pretrained_ds,\n", " epochs=8,\n", " validation_data=val_pretrained_ds,\n", " callbacks=[\n", " callbacks.EarlyStopping(monitor=\"val_loss\", patience=2, restore_best_weights=True),\n", " callbacks.ReduceLROnPlateau(monitor=\"val_loss\", factor=0.5, patience=1, min_lr=1e-6)\n", " ],\n", " verbose=1\n", " )\n", " pretrained_train_time += time.perf_counter() - start_fine\n", " plot_training_history(pretrained_fine_tune_history, \"ResNet50V2 Fine-Tuning\")\n" ] }, { "cell_type": "code", "execution_count": null, "id": "d179e666", "metadata": {}, "outputs": [], "source": [ "if USE_PRETRAINED_BASELINE:\n", " start = time.perf_counter()\n", " pretrained_probs = pretrained_baseline.predict(test_pretrained_ds, verbose=0)\n", " pretrained_pred_time = time.perf_counter() - start\n", " pretrained_pred = np.argmax(pretrained_probs, axis=1)\n", "\n", " pretrained_val_loss, pretrained_val_acc = pretrained_baseline.evaluate(val_pretrained_ds, verbose=0)\n", " pretrained_test_loss, pretrained_test_acc = pretrained_baseline.evaluate(test_pretrained_ds, verbose=0)\n", " pretrained_test_f1 = f1_score(y_test_enc, pretrained_pred, average=\"macro\")\n", "\n", " results.append({\n", " \"model\": \"ResNet50V2\",\n", " \"type\": \"Advanced pretrained baseline\",\n", " \"parameters\": pretrained_baseline.count_params(),\n", " \"train_time_sec\": pretrained_train_time,\n", " \"prediction_time_sec\": pretrained_pred_time,\n", " \"val_accuracy\": pretrained_val_acc,\n", " \"test_accuracy\": pretrained_test_acc,\n", " \"test_macro_f1\": pretrained_test_f1\n", " })\n", "\n", " print(f\"ResNet50V2 validation accuracy: {pretrained_val_acc:.4f}\")\n", " print(f\"ResNet50V2 test accuracy: {pretrained_test_acc:.4f}\")\n", " print(f\"ResNet50V2 test macro F1: {pretrained_test_f1:.4f}\")\n", " print(\"\\nClassification report:\")\n", " print(classification_report(y_test_enc, pretrained_pred, target_names=encoded_class_names))\n", " plot_confusion_matrix(y_test_enc, pretrained_pred, \"ResNet50V2 Baseline Confusion Matrix\")\n", " plot_confusion_matrix(y_test_enc, pretrained_pred, \"ResNet50V2 Baseline Confusion Matrix\", normalize=True)\n", " display(summarize_confusion_errors(y_test_enc, pretrained_pred))\n" ] }, { "cell_type": "markdown", "id": "a814ce31", "metadata": {}, "source": [ "## 9. CNN Interpretability with Grad-CAM\n", "\n", "Traditional feature importance is not directly available for CNNs. Instead, we can use a visual explanation method such as Grad-CAM.\n", "\n", "Grad-CAM highlights image regions that influenced the model prediction. For emotion recognition, useful regions should often include the mouth, eyes, and eyebrows." ] }, { "cell_type": "code", "execution_count": null, "id": "936629ce", "metadata": {}, "outputs": [], "source": [ "def make_gradcam_heatmap(img_array, model, last_conv_layer_name=None, pred_index=None):\n", " \"\"\"Compute a Grad-CAM heatmap for one image and one predicted class.\"\"\"\n", " if last_conv_layer_name is None:\n", " # Use the last convolutional layer automatically so the function still works after architecture changes.\n", " last_conv_layer = next(\n", " (layer for layer in reversed(model.layers) if isinstance(layer, tf.keras.layers.Conv2D)),\n", " None\n", " )\n", " if last_conv_layer is None:\n", " raise ValueError(\"No Conv2D layer found in the model for Grad-CAM.\")\n", " last_conv_layer_name = last_conv_layer.name\n", "\n", " grad_model = tf.keras.models.Model(\n", " [model.inputs],\n", " [model.get_layer(last_conv_layer_name).output, model.output]\n", " )\n", "\n", " with tf.GradientTape() as tape:\n", " conv_outputs, predictions = grad_model(img_array)\n", " if pred_index is None:\n", " pred_index = tf.argmax(predictions[0])\n", " class_channel = predictions[:, pred_index]\n", "\n", " # Average gradients tell us which feature maps mattered most for the class score.\n", " grads = tape.gradient(class_channel, conv_outputs)\n", " pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n", " conv_outputs = conv_outputs[0]\n", " heatmap = conv_outputs @ pooled_grads[..., tf.newaxis]\n", " heatmap = tf.squeeze(heatmap)\n", " heatmap = tf.maximum(heatmap, 0) / (tf.math.reduce_max(heatmap) + 1e-8)\n", " return heatmap.numpy()\n", "\n", "\n", "def show_gradcam_examples(n=5):\n", " indices = np.random.choice(len(X_test_cnn), size=min(n, len(X_test_cnn)), replace=False)\n", " fig, axes = plt.subplots(len(indices), 3, figsize=(9, 3 * len(indices)))\n", " if len(indices) == 1:\n", " axes = np.expand_dims(axes, axis=0)\n", "\n", " for row, idx in enumerate(indices):\n", " img = X_test_cnn[idx:idx+1]\n", " probs = custom_cnn.predict(img, verbose=0)[0]\n", " pred_idx = int(np.argmax(probs))\n", " true_idx = int(y_test_enc[idx])\n", " heatmap = make_gradcam_heatmap(img, custom_cnn, pred_index=pred_idx)\n", " heatmap_resized = Image.fromarray(np.uint8(255 * heatmap)).resize(IMG_SIZE)\n", "\n", " axes[row, 0].imshow(X_test[idx], cmap=\"gray\")\n", " axes[row, 0].set_title(f\"True: {encoded_class_names[true_idx]}\")\n", " axes[row, 0].axis(\"off\")\n", "\n", " axes[row, 1].imshow(heatmap_resized, cmap=\"jet\")\n", " axes[row, 1].set_title(\"Grad-CAM\")\n", " axes[row, 1].axis(\"off\")\n", "\n", " axes[row, 2].imshow(X_test[idx], cmap=\"gray\")\n", " axes[row, 2].imshow(heatmap_resized, cmap=\"jet\", alpha=0.45)\n", " axes[row, 2].set_title(f\"Pred: {encoded_class_names[pred_idx]}\")\n", " axes[row, 2].axis(\"off\")\n", "\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "show_gradcam_examples(n=5)\n" ] }, { "cell_type": "markdown", "id": "e227a5a1", "metadata": {}, "source": [ "## 10. Model Complexity and Performance Comparison\n", "\n", "This table compares model performance and complexity. The final interpretation should consider both accuracy and cost.\n", "\n", "A model is not automatically better only because it has higher accuracy. Runtime, number of parameters, interpretability, and risk of overfitting also matter." ] }, { "cell_type": "code", "execution_count": null, "id": "5351a85b", "metadata": {}, "outputs": [], "source": [ "results_df = pd.DataFrame(results)\n", "results_df = results_df.sort_values(\"test_macro_f1\", ascending=False)\n", "results_df" ] }, { "cell_type": "code", "execution_count": null, "id": "1ed49d87", "metadata": {}, "outputs": [], "source": [ "plot_df = results_df.copy()\n", "plot_df[\"test_accuracy\"] = plot_df[\"test_accuracy\"].astype(float)\n", "plot_df[\"test_macro_f1\"] = plot_df[\"test_macro_f1\"].astype(float)\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(13, 4))\n", "sns.barplot(data=plot_df, x=\"model\", y=\"test_accuracy\", ax=axes[0], palette=\"viridis\")\n", "axes[0].set_title(\"Test Accuracy by Model\")\n", "axes[0].tick_params(axis=\"x\", rotation=25)\n", "axes[0].set_ylim(0, 1)\n", "\n", "sns.barplot(data=plot_df, x=\"model\", y=\"test_macro_f1\", ax=axes[1], palette=\"mako\")\n", "axes[1].set_title(\"Macro F1 by Model\")\n", "axes[1].tick_params(axis=\"x\", rotation=25)\n", "axes[1].set_ylim(0, 1)\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "e7064824", "metadata": {}, "source": [ "### Main Confusion Patterns\n", "\n", "This table summarizes the most common wrong emotion pairs. It is useful for interpreting the confusion matrices and for deciding where more data, augmentation, or class-specific tuning may help.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b33a48f6", "metadata": {}, "outputs": [], "source": [ "prediction_sets = [\n", " (\"Dummy Baseline\", dummy_pred),\n", " (\"Logistic Regression\", log_reg_pred),\n", " (\"Random Forest\", rf_pred),\n", " (\"Custom CNN\", cnn_pred),\n", "]\n", "\n", "if \"pretrained_pred\" in globals():\n", " prediction_sets.append((\"ResNet50V2\", pretrained_pred))\n", "\n", "error_tables = []\n", "for model_name, predictions in prediction_sets:\n", " model_errors = summarize_confusion_errors(y_test_enc, predictions, top_n=5).copy()\n", " model_errors.insert(0, \"model\", model_name)\n", " error_tables.append(model_errors)\n", "\n", "confusion_error_summary = pd.concat(error_tables, ignore_index=True)\n", "confusion_error_summary[\"share_of_true_class\"] = confusion_error_summary[\"share_of_true_class\"].round(3)\n", "confusion_error_summary\n" ] }, { "cell_type": "markdown", "id": "bde16256", "metadata": {}, "source": [ "## 11. Overfitting and Underfitting Check for All Models\n", "\n", "The assignment asks to verify overfitting and underfitting issues. For neural networks, this is usually checked with training and validation curves. For traditional machine learning models, it is checked by comparing training, validation, and test performance.\n", "\n", "Interpretation rule:\n", "\n", "- **Overfitting:** training accuracy is much higher than validation/test accuracy.\n", "- **Underfitting:** training accuracy and validation/test accuracy are both low.\n", "- **Reasonable fit:** training and validation/test accuracy are close, and performance is acceptable for the task.\n", "\n", "The table below computes this comparison for all models.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b97af619", "metadata": {}, "outputs": [], "source": [ "def diagnose_fit(train_acc, val_acc, test_acc):\n", " gap = train_acc - val_acc\n", " if gap >= 0.12:\n", " return \"Overfitting risk: training accuracy is much higher than validation accuracy.\"\n", " if train_acc < 0.40 and val_acc < 0.40:\n", " return \"Underfitting / limited learning: both training and validation accuracy are low.\"\n", " if val_acc < 0.45 and test_acc < 0.45:\n", " return \"Mild underfitting: model learns useful patterns, but performance is still limited.\"\n", " return \"Reasonable fit: training and validation performance are close.\"\n", "\n", "fit_rows = []\n", "\n", "traditional_models = [\n", " (\"Logistic Regression\", log_reg, X_train_flat, X_val_flat, X_test_flat),\n", " (\"Random Forest\", random_forest, X_train_flat, X_val_flat, X_test_flat),\n", "]\n", "\n", "for name, model, Xtr, Xv, Xte in traditional_models:\n", " train_acc = accuracy_score(y_train, model.predict(Xtr))\n", " val_acc = accuracy_score(y_val, model.predict(Xv))\n", " test_acc = accuracy_score(y_test_enc, model.predict(Xte))\n", " fit_rows.append({\n", " \"model\": name,\n", " \"train_accuracy\": train_acc,\n", " \"validation_accuracy\": val_acc,\n", " \"test_accuracy\": test_acc,\n", " \"train_val_gap\": train_acc - val_acc,\n", " \"diagnosis\": diagnose_fit(train_acc, val_acc, test_acc)\n", " })\n", "\n", "cnn_train_loss, cnn_train_acc = custom_cnn.evaluate(X_train_cnn, y_train, verbose=0)\n", "cnn_val_loss, cnn_val_acc = custom_cnn.evaluate(X_val_cnn, y_val, verbose=0)\n", "cnn_test_loss, cnn_test_acc = custom_cnn.evaluate(X_test_cnn, y_test_enc, verbose=0)\n", "fit_rows.append({\n", " \"model\": \"Custom CNN\",\n", " \"train_accuracy\": cnn_train_acc,\n", " \"validation_accuracy\": cnn_val_acc,\n", " \"test_accuracy\": cnn_test_acc,\n", " \"train_val_gap\": cnn_train_acc - cnn_val_acc,\n", " \"diagnosis\": diagnose_fit(cnn_train_acc, cnn_val_acc, cnn_test_acc)\n", "})\n", "\n", "if \"pretrained_baseline\" in globals() and \"train_pretrained_eval_ds\" in globals():\n", " pretrained_train_loss, pretrained_train_acc = pretrained_baseline.evaluate(train_pretrained_eval_ds, verbose=0)\n", " pretrained_val_loss, pretrained_val_acc = pretrained_baseline.evaluate(val_pretrained_ds, verbose=0)\n", " pretrained_test_loss, pretrained_test_acc = pretrained_baseline.evaluate(test_pretrained_ds, verbose=0)\n", " fit_rows.append({\n", " \"model\": \"ResNet50V2\",\n", " \"train_accuracy\": pretrained_train_acc,\n", " \"validation_accuracy\": pretrained_val_acc,\n", " \"test_accuracy\": pretrained_test_acc,\n", " \"train_val_gap\": pretrained_train_acc - pretrained_val_acc,\n", " \"diagnosis\": diagnose_fit(pretrained_train_acc, pretrained_val_acc, pretrained_test_acc)\n", " })\n", "\n", "fit_diagnosis_df = pd.DataFrame(fit_rows)\n", "fit_diagnosis_df[[\"train_accuracy\", \"validation_accuracy\", \"test_accuracy\", \"train_val_gap\"]] = fit_diagnosis_df[[\"train_accuracy\", \"validation_accuracy\", \"test_accuracy\", \"train_val_gap\"]].round(4)\n", "fit_diagnosis_df\n" ] }, { "cell_type": "markdown", "id": "e31b974b", "metadata": {}, "source": [ "### Overfitting / Underfitting Interpretation\n", "\n", "Use the table above together with the CNN/ResNet learning curves.\n", "\n", "In the saved output:\n", "\n", "- **Logistic Regression** has a training-validation gap of about `0.172`, so it shows overfitting even though its overall score is low. Flattened pixels are not expressive enough for this image task.\n", "- **Random Forest** has the largest overfitting problem, with training accuracy around `0.918` but validation accuracy around `0.420`. It memorizes many training pixel patterns but does not generalize well.\n", "- **Custom CNN** has a small train-validation gap of about `0.033` and the best test performance, so it is the best-balanced model in this run.\n", "- **ResNet50V2** also has a reasonable train-validation gap of about `0.071`, but its final test accuracy and macro F1 are lower than the custom CNN.\n" ] }, { "cell_type": "markdown", "id": "992c0391", "metadata": {}, "source": [ "## 12. Results and Key Findings\n", "\n", "The saved output shows the following ranking by test macro F1:\n", "\n", "| Model | Validation accuracy | Test accuracy | Test macro F1 |\n", "|---|---:|---:|---:|\n", "| Custom CNN | 0.587 | 0.595 | 0.552 |\n", "| ResNet50V2 | 0.536 | 0.532 | 0.472 |\n", "| Random Forest | 0.420 | 0.429 | 0.412 |\n", "| Logistic Regression | 0.313 | 0.321 | 0.295 |\n", "| Dummy Baseline | 0.251 | 0.247 | 0.057 |\n", "\n", "### Key Findings\n", "\n", "1. **The custom CNN is the best model in this run.** \n", " It achieves the highest test accuracy and macro F1 while keeping the train-validation gap small. This supports using it as the main model.\n", "\n", "2. **The dummy baseline confirms the task is non-trivial.** \n", " Always predicting the majority class reaches only about `24.7%` test accuracy and `0.057` macro F1. All real models beat this baseline.\n", "\n", "3. **Traditional ML models are limited on raw image pixels.** \n", " Logistic Regression and Random Forest use flattened images, so they lose spatial structure. Random Forest improves over Logistic Regression, but it strongly overfits.\n", "\n", "4. **ResNet50V2 is useful but not the winner here.** \n", " The pretrained model beats the traditional ML models, but it does not beat the custom CNN. Transfer learning is not automatically better when the target data is small, grayscale, and low-resolution.\n", "\n", "5. **The hardest errors are between visually similar emotions.** \n", " The confusion-pair table shows frequent mistakes such as `sad -> neutral`, `fear -> sad`, `fear -> neutral`, and `neutral -> sad`. These classes have overlapping facial cues.\n", "\n", "### Causal Explanation\n", "\n", "The custom CNN performs best because it preserves spatial image structure and learns local facial patterns directly from FER-2013. Traditional flattened-pixel models lose spatial relationships, and ResNet50V2 transfers from ImageNet features that are not perfectly matched to low-resolution grayscale facial expressions.\n" ] }, { "cell_type": "markdown", "id": "210893ce", "metadata": {}, "source": [ "## 13. Actionable Insights\n", "\n", "The results support these decisions:\n", "\n", "- The custom CNN should be presented as the main final model because it has the best score and a reasonable fit.\n", "- ResNet50V2 should be presented as an advanced baseline, not as the final winner.\n", "- Random Forest is useful for feature importance, but its large train-validation gap means it should not be chosen as the final model.\n", "- Logistic Regression is useful as a simple classical comparison, but it is too limited for this image task.\n", "- Future improvements should focus on confusing classes such as `sad`, `neutral`, and `fear`.\n", "- If Grad-CAM shows attention outside important face regions, preprocessing such as face alignment or better cropping should be considered.\n" ] }, { "cell_type": "markdown", "id": "f8673418", "metadata": {}, "source": [ "## 14. Practical Outcomes\n", "\n", "Possible real-world applications include:\n", "\n", "- human-computer interaction,\n", "- educational feedback systems,\n", "- customer experience analysis,\n", "- entertainment and game interfaces,\n", "- assistive technologies.\n", "\n", "However, this model should not be used alone for high-stakes decisions. Facial expression is not the same as true internal emotion. A person may smile while uncomfortable, or appear neutral while experiencing stress." ] }, { "cell_type": "markdown", "id": "c25485a6", "metadata": {}, "source": [ "## 15. Ethical Discussion\n", "\n", "Facial emotion recognition has important ethical risks.\n", "\n", "### Privacy\n", "\n", "Face images are sensitive biometric data. Users should give informed consent before their images are collected or processed.\n", "\n", "### Bias and Fairness\n", "\n", "The model may perform differently across age groups, genders, ethnicities, cultures, lighting conditions, and camera quality. If the training data is not representative, predictions may be unfair.\n", "\n", "### Emotional Ambiguity\n", "\n", "Emotion labels are subjective. The same facial expression can have different meanings depending on context, culture, and individual behavior.\n", "\n", "### Misuse Risk\n", "\n", "Emotion recognition should not be used for surveillance, hiring, policing, grading, or medical decisions without strict validation and human oversight.\n", "\n", "### Responsible Use Recommendation\n", "\n", "Use this technology only in low-risk settings, with transparency, consent, clear limitations, and the option for users to opt out." ] }, { "cell_type": "markdown", "id": "085e580c", "metadata": {}, "source": [ "## 16. Conclusion\n", "\n", "This project compared a dummy sanity baseline, three own models, and one advanced pretrained baseline for FER-2013 facial emotion classification.\n", "\n", "The best model in the saved run is the **custom residual CNN**, with about `59.5%` test accuracy and `0.552` test macro F1. It outperforms ResNet50V2, Random Forest, Logistic Regression, and the dummy baseline. It also has a small train-validation gap, so it generalizes better than the traditional models.\n", "\n", "The ResNet50V2 baseline reaches about `53.2%` test accuracy and `0.472` macro F1. This is better than the traditional ML models, but lower than the custom CNN. The result shows that transfer learning is helpful, but not automatically superior for small grayscale emotion images.\n", "\n", "Traditional ML models provide useful comparison points. Logistic Regression is simple but weak, while Random Forest improves the score but strongly overfits. The dummy baseline confirms that all trained models learn meaningful information beyond the majority class.\n", "\n", "The most important lesson is that model performance must be evaluated together with complexity, interpretability, runtime, overfitting behavior, class-level errors, and ethical risk.\n", "\n", "### Future Improvements\n", "\n", "- tune CNN depth, dropout, and learning rate more carefully,\n", "- compare class weights against no class weights for the CNN,\n", "- test conservative ResNet fine-tuning settings such as 10, 20, or 30 unfrozen layers,\n", "- try stronger but still realistic augmentation,\n", "- investigate the most confused classes: `sad`, `neutral`, and `fear`,\n", "- test on external real-world images,\n", "- improve interpretability with more Grad-CAM examples,\n", "- consider face alignment or face-cropping preprocessing if Grad-CAM shows attention outside the face.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }