{
 "nbformat": 4,
 "nbformat_minor": 0,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Credit Card Approval Prediction Project"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Machine Learning Course Project\n",
    "The purpose of this project is to build a predictive model that determines whether an applicant should be approved for a credit card. This involves analyzing various attributes of applicants to predict the likelihood of approval or rejection, aiding decision-making processes in financial institutions. The key objectives are:\n",
    "\n",
    "- Risk Mitigation: Minimize risks by identifying applicants who are less likely to meet repayment obligations.\n",
    "- Profit Maximization: Avoid rejecting potentially profitable clients to optimize revenue opportunities.\n",
    "\n",
    "The focus is to balance these objectives while providing accurate and actionable insights for real-world credit approval scenarios.\n",
    "\n",
    "\n",
    "Various machine learning models were implemented, including Logistic Regression, Support Vector Machines (SVM), Decision Trees, Random Forests, and Neural Networks. The models were trained using a well-structured preprocessing pipeline, addressing missing values, categorical encoding, feature scaling, and data imbalance. Performance was assessed using key metrics such as Precision, F1-score, and AUC-ROC. The results indicate that Logistic Regression, SVM, and Decision Trees performed exceptionally well, achieving near-perfect classification accuracy. The study highlights the importance of data preprocessing and hyperparameter tuning in optimizing predictive performance."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 1. Introduction and Setup"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Loading packages"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.preprocessing import StandardScaler, LabelEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import precision_score, f1_score, roc_auc_score, confusion_matrix\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "\"\"\"\n",
    "Project Overview:\n",
    "- Purpose: Build a predictive model for credit card approval\n",
    "- Key Objectives: Risk Mitigation and Profit Maximization\n",
    "- Main Metrics: Precision, F1-score, and AUC-ROC\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 2. Data Loading and Exploration"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# Load the dataset\n",
    "data = pd.read_csv(\"Application_Data.csv\")"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Initial data exploration"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "print(\"Dataset Shape:\", data.shape)\n",
    "print(\"\\nFirst few rows:\")\n",
    "display(data.head())\n",
    "print(\"\\nDataset Info:\")\n",
    "display(data.info())"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see few of the features types are not a numerical numbers. We will deal with them using one-hot encoding."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Basic statistical summary\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "print(\"\\nNumerical Features Summary:\")\n",
    "display(data.describe())"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3. Data Analysis and Visualization"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "sns.heatmap(data.isnull(), yticklabels=False, cbar=True, cmap='viridis')\n",
    "plt.title('Missing Values in Dataset')\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "No missing values.\n",
    "\n",
    "**Now let's see some visualizations of data, to get some understanding of it**"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# Checking for the dataset's column names\n",
    "print(\"Column names in the dataset:\")\n",
    "print(data.columns)\n",
    "\n",
    "# Visualizing the distribution of numerical features\n",
    "numerical_features = ['Total_Income', 'Total_Children', 'Applicant_Age', 'Years_of_Working',\n",
    "                      'Total_Bad_Debt', 'Total_Good_Debt']\n",
    "\n",
    "# Creating histograms for numerical features\n",
    "plt.figure(figsize=(15, 10))\n",
    "for i, col in enumerate(numerical_features, 1):\n",
    "    plt.subplot(2, 3, i)\n",
    "    sns.histplot(data[col], kde=True, color='blue')\n",
    "    plt.title(f'Distribution of {col}')\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Visualizing the categorical features using bar plots\n",
    "categorical_features = ['Applicant_Gender', 'Owned_Car', 'Owned_Realty', 'Income_Type',\n",
    "                        'Education_Type', 'Family_Status', 'Housing_Type']\n",
    "\n",
    "plt.figure(figsize=(15, 15))\n",
    "for i, col in enumerate(categorical_features, 1):\n",
    "    plt.subplot(4, 2, i)\n",
    "    sns.countplot(data=data, x=col, hue=col, palette='viridis', legend=False)\n",
    "    plt.title(f'Distribution of {col}')\n",
    "    plt.xticks(rotation=45)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Analyze target variable distribution"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "total_applications = len(data)\n",
    "approved_applications = data['Status'].sum()  # Assuming '1' indicates approval\n",
    "approved_percentage = (approved_applications / total_applications) * 100\n",
    "\n",
    "# Display the result\n",
    "print(f\"Total Applications: {total_applications}\")\n",
    "print(f\"Approved Applications: {approved_applications}\")\n",
    "print(f\"Percentage of Approved Applications: {approved_percentage:.2f}%\")"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "As we can see, the data is very unbalanced, we will have to do something with it before training our model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 4. Data Preprocessing"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Handling categorical data\n",
    "- Encoding categorical variables (one-hot encoding).\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "print(data.columns)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Helps avoid multicollinearity in models like Logistic Regression, where linear dependencies can degrade performance.\n",
    "Reduces the dimensionality of the dataset slightly, which is beneficial if you have many categorical variables with multiple categories."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# One-hot encoding with dropping the first category\n",
    "data_encoded = pd.get_dummies(\n",
    "    data[['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type', 'Job_Title']],\n",
    "    drop_first=True\n",
    ")\n",
    "\n",
    "# Add the encoded columns back to the original dataset, dropping the original categorical columns\n",
    "data_reduced = pd.concat([data.drop(columns=['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type', 'Job_Title']),\n",
    "                  data_encoded], axis=1)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "print(data_reduced.shape)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Might be useful for tree-based models (e.g., Decision Trees, Random Forest) since they are not affected by multicollinearity."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# One-hot encoding without dropping the first category\n",
    "data_encoded = pd.get_dummies(\n",
    "    data[['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type', 'Job_Title']]\n",
    ")\n",
    "\n",
    "# Add the encoded columns back to the original dataset, dropping the original categorical columns\n",
    "data_forest = pd.concat([data.drop(columns=['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type', 'Job_Title']),\n",
    "                  data_encoded], axis=1)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "print(data_forest.shape)"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "# Correlation of \"Status\" with numeric features\n",
    "correlations = data_forest.corr()  # Compute pairwise correlation\n",
    "status_correlations = correlations['Status'].sort_values(ascending=False)\n",
    "print(\"\\nCorrelation of 'Status' with other numeric variables:\")\n",
    "print(status_correlations)"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Job title means the least, so we will drop this collumn."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# One-hot encoding with dropping the first category\n",
    "data_encoded = pd.get_dummies(\n",
    "    data[['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type']],\n",
    "    drop_first=True\n",
    ")\n",
    "\n",
    "# Add the encoded columns back to the original dataset, dropping the original categorical columns\n",
    "data_reduced = pd.concat([data.drop(columns=['Applicant_Gender', 'Income_Type', 'Education_Type', 'Family_Status', 'Housing_Type']),\n",
    "                  data_encoded], axis=1)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Partitioning dataset into a separate training and test set"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Partitioning for training logistic regression, SVM"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = data_reduced.drop(['Status', 'Applicant_ID','Job_Title'], axis=1)\n",
    "y = data_reduced['Status']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.4, random_state=1, stratify=y)"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Partitioning for training three-based models"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "\"\"\"\n",
    "X_f = data_forest.drop(['Status', 'Applicant_ID'], axis=1)\n",
    "y_f = data_forest['Status']\n",
    "\n",
    "X_train_f, X_test_f, y_train_f, y_test_f =\\\n",
    "     train_test_split(X_f, y_f,\n",
    "                      test_size=0.4,\n",
    "                      random_state=0,\n",
    "                      stratify=y_f)\n",
    "\"\"\""
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Bringing features onto the same scale"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Decision trees and random forests are two of the very few machine learning algorithms where w don't need to worry about feature scalling. However, the majority algorigms behave much better if features are one the same scale. Since we will be using logistic regression, neural networks and SVM, we will now procedure standarization, so they can perform better."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now preprocess our training and test data separetely, so we do not have data leakage. That way we won't influence our training data by test data\n",
    "- Feature scaling using standardization.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "columns_to_standardize = ['Total_Income', 'Applicant_Age', 'Years_of_Working', 'Total_Bad_Debt', 'Total_Good_Debt']\n",
    "\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Fit the scaler on the training data (only the specified columns) and transform the training data\n",
    "X_train_scaled = X_train.copy()\n",
    "X_train_scaled[columns_to_standardize] = scaler.fit_transform(X_train[columns_to_standardize])\n",
    "\n",
    "# Transform the test data (only the specified columns) using the same scaler\n",
    "X_test_scaled = X_test.copy()\n",
    "X_test_scaled[columns_to_standardize] = scaler.transform(X_test[columns_to_standardize])"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Dealing with imbalanced data"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "For this specific credit card approval problem, we will use upsampling of the minor class"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "total_applications_train = len(y_train)  # Get total number of applications in training data\n",
    "sum_status = y_train.sum()  # Get total number of approved applications (assuming '1' represents approval)\n",
    "approved_percentage_train = (sum_status / total_applications_train) * 100  # Calculate percentage\n",
    "\n",
    "print(f\"Total Applications in Training Data: {total_applications_train}\")\n",
    "print(f\"Approved Applications in Training Data: {sum_status}\")\n",
    "print(f\"Percentage of Approved Applications in Training Data: {approved_percentage_train:.2f}%\")"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "- Addressing data imbalance using SMOTETomek.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# undersample only the training data\n",
    "from imblearn.under_sampling import RandomUnderSampler\n",
    "from imblearn.combine import SMOTETomek\n",
    "\n",
    "# 1. Better Sampling Approach - Using SMOTETomek instead of just undersampling\n",
    "# This combines SMOTE with Tomek links to create better quality samples\n",
    "smote_tomek = SMOTETomek(random_state=42, sampling_strategy=0.4)  # Create 50% of majority class\n",
    "X_train_resampled, y_train_resampled = smote_tomek.fit_resample(X_train_scaled, y_train)\n",
    "\n",
    "# Test set remains with original distribution\n",
    "# This way we can evaluate how our model performs on real-world data"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "total_applications_train = len(y_train_resampled)  # Get total number of applications in training data\n",
    "sum_status = y_train_resampled.sum()  # Get total number of approved applications (assuming '1' represents approval)\n",
    "approved_percentage_train = (sum_status / total_applications_train) * 100  # Calculate percentage\n",
    "\n",
    "print(f\"Total Applications in Training Data: {total_applications_train}\")\n",
    "print(f\"Approved Applications in Training Data: {sum_status}\")\n",
    "print(f\"Percentage of Approved Applications in Training Data: {approved_percentage_train:.2f}%\")"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Training models"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Evaluation with logistic regression"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# Part 1: Basic Logistic Regression with different C values\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Initialize with high C value for less regularization\n",
    "lr = LogisticRegression(C=100.0, random_state=1, class_weight='balanced', solver='lbfgs')\n",
    "lr.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Part 2: Testing different regularization strengths with color palette\n",
    "weights, params = [], []\n",
    "for c in np.arange(-5, 5):  # Using range -5 to 5 for log scale of C\n",
    "    lr = LogisticRegression(C=10.**c, random_state=1, class_weight='balanced', solver='lbfgs')\n",
    "    lr.fit(X_train_resampled, y_train_resampled)\n",
    "    weights.append(lr.coef_[0])\n",
    "    params.append(10.**c)\n",
    "\n",
    "weights = np.array(weights)\n",
    "\n",
    "# Define a color palette with enough colors for all features\n",
    "num_features = X_train_resampled.shape[1]\n",
    "colors = plt.cm.viridis(np.linspace(0, 1, num_features))  # Example using viridis\n",
    "\n",
    "# Plot the regularization path with colors\n",
    "plt.figure(figsize=(12, 8))  # Increase figure size for better visibility\n",
    "for column, feature in enumerate(X_train_resampled.columns):\n",
    "    plt.plot(params, weights[:, column],\n",
    "             label=feature,\n",
    "             linestyle='-',\n",
    "             color=colors[column])  # Assign color from palette\n",
    "\n",
    "plt.ylabel('Weight Coefficient')\n",
    "plt.xlabel('C (Regularization Parameter)')\n",
    "plt.xscale('log')\n",
    "plt.title('Feature Weights vs Regularization Strength')\n",
    "\n",
    "# Enhancements for better visibility\n",
    "plt.legend(loc='upper left', bbox_to_anchor=(1.05, 1), fontsize='small')  # Adjust legend size\n",
    "# plt.grid(True, alpha=0.5)  # Add grid for better readability (optional)\n",
    "# plt.xlim([params[0], params[-1]])  # Adjust x-axis limits if needed\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "plt.savefig('regularization_path.png', dpi=300)  # Save as PNG with 300 DPI\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's observe this Regularization Path plot:\n",
    "\n",
    "Stable region: The weights of important features are relatively stable between C values of 0.1 and 10.\n",
    "Feature importance: The features you expect to be important have significant weights in this stable region.\n",
    "Model performance: Using cross-validation, you find that the model achieves the highest F1-score with a C value of 1.\n",
    "In this case, a C value of 1 would be a good starting point. You might also experiment with values slightly above or below 1 (e.g., 0.5, 2) to see if you can get any further improvement in performance.\n",
    "\n",
    "Important Notes:\n",
    "\n",
    "The \"best\" C value depends on your specific dataset, problem, and the chosen evaluation metrics.\n",
    "It's essential to use a validation set or cross-validation to get a realistic estimate of the model's generalization performance.\n",
    "Be aware of the trade-off between model complexity and generalization. Too little regularization might lead to overfitting, while too much regularization could result in underfitting."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create logistic regression model with class weight adjustment\n",
    "lr_model = LogisticRegression(C=1000,\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    max_iter=1000  # Increasing max_iter due to potential convergence needs\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "lr_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions (class labels and probabilities)\n",
    "y_pred = lr_model.predict(X_test_scaled)\n",
    "y_pred_prob = lr_model.predict_proba(X_test_scaled)[:, 1]  # Probability for the positive class\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "- True Negatives (TN): 48 (Top-left). The model correctly predicted 48 'Rejected' applications as 'Rejected'.\n",
    "- False Positives (FP): 0 (Top-right). The model incorrectly predicted 0 'Rejected' applications as 'Approved'.\n",
    "- False Negatives (FN): 4 (Bottom-left). The model incorrectly predicted 4 'Approved' applications as 'Rejected'.\n",
    "- True Positives (TP): 10000 (Bottom-right). The model correctly predicted 10000 'Approved' applications as 'Approved'.\n",
    "\n",
    "This is what we expect from our algorithm. It deals well with interpreting who shouldn't get a credit card. The four misinterpreted examples, where someone should get credit card are not that important."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Hyperparameter tuning to optimize performance done for this algorithm:\n",
    "- choosing best value fo C\n",
    "- choosing class_weight='balanced', due to imbalanced data\n",
    "- changing a number for max iter\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Training with SVM"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create SVM model with probability estimates\n",
    "svm_model = SVC(C=9.522017556848526,\n",
    "    kernel='linear',  # Radial Basis Function kernel\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    probability=True,  # Required for ROC curve\n",
    "                )\n",
    "\n",
    "# Train the model\n",
    "svm_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = svm_model.predict(X_test_scaled)\n",
    "y_pred_prob = svm_model.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.svm import SVC\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Define the SVM model\n",
    "svm = SVC()\n",
    "\n",
    "# Define the hyperparameter distributions\n",
    "param_distributions = {\n",
    "    'C': uniform(0.1, 10),  # Węższy zakres\n",
    "    'gamma': [0.01, 0.1, 'scale'],  # Mniej opcji\n",
    "    'kernel': ['rbf', 'linear']  # Mniej kernelów\n",
    "}\n",
    "random_search = RandomizedSearchCV(estimator=svm, param_distributions=param_distributions, n_iter=20, cv=3)\n",
    "\n",
    "\n",
    "# Perform Randomized Search\n",
    "random_search = RandomizedSearchCV(estimator=svm, param_distributions=param_distributions, n_iter=50, scoring='accuracy', cv=3, verbose=1, n_jobs=-1, random_state=42)\n",
    "random_search.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Output the best parameters and score\n",
    "print(\"Best Hyperparameters:\", random_search.best_params_)\n",
    "print(\"Best Cross-Validation Accuracy:\", random_search.best_score_)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "I applicated C and kernel to linear to my model before this text"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Decision tree"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create Decision Tree model\n",
    "dt_model = DecisionTreeClassifier(\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    max_depth=11  # Prevent overfitting\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "dt_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = dt_model.predict(X_test_scaled)\n",
    "y_pred_prob = dt_model.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "For current depth I got the best results for me, also the precision is the biggest"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn import tree\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(15, 10))  # Adjust figure size for initial clarity\n",
    "tree.plot_tree(dt_model,\n",
    "               max_depth=3,  # Limit depth for initial visualization\n",
    "               feature_names=X_train_resampled.columns,\n",
    "               class_names=['Rejected', 'Approved'],\n",
    "               filled=True)\n",
    "\n",
    "plt.savefig('decision_tree.png', dpi=300)  # Save as PNG with higher resolution\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now lets do some hyperparameter tuning for decision trees."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import randint\n",
    "\n",
    "param_dist = {\n",
    "    'max_depth': [3, 5, 10, None],\n",
    "    'min_samples_split': randint(2, 20),\n",
    "    'min_samples_leaf': randint(1, 20),\n",
    "    'criterion': ['gini', 'entropy']\n",
    "}\n",
    "\n",
    "random_search = RandomizedSearchCV(estimator=dt_model, param_distributions=param_dist, n_iter=50, cv=5, scoring='accuracy', n_jobs=-1, random_state=42)\n",
    "random_search.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "print(\"Best parameters:\", random_search.best_params_)\n",
    "print(\"Best score:\", random_search.best_score_)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "#implementation:\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create Decision Tree model\n",
    "dt_model = DecisionTreeClassifier(\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    max_depth=10,  # Prevent overfitting\n",
    "    criterion= 'entropy',\n",
    "    min_samples_leaf= 3,\n",
    "    min_samples_split= 2\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "dt_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = dt_model.predict(X_test_scaled)\n",
    "y_pred_prob = dt_model.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Quite good results."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Random forest"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create Random Forest model\n",
    "rf_model = RandomForestClassifier(\n",
    "    n_estimators=100,  # Number of trees\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    max_depth=11  # Prevent overfitting\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "rf_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = rf_model.predict(X_test_scaled)\n",
    "y_pred_prob = rf_model.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Lets do hyperparameter tuining"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import randint\n",
    "\n",
    "# Instantiate the Random Forest model\n",
    "rf = RandomForestClassifier(random_state=42)\n",
    "\n",
    "# Define a smaller hyperparameter distribution\n",
    "param_dist = {\n",
    "    'n_estimators': [50, 100, 200],  # Fewer options for the number of trees\n",
    "    'max_depth': [10, 20, None],  # Limited depth options\n",
    "    'min_samples_split': [2, 5, 10],  # A smaller range for splitting\n",
    "    'min_samples_leaf': [1, 2, 4],  # A smaller range for leaf size\n",
    "    'max_features': ['sqrt', 'log2'],  # A couple of feature selection strategies\n",
    "}\n",
    "\n",
    "# Perform randomized search with fewer iterations\n",
    "random_search = RandomizedSearchCV(\n",
    "    estimator=rf,\n",
    "    param_distributions=param_dist,\n",
    "    n_iter=10,  # Reduced number of iterations\n",
    "    cv=3,  # Use 3-fold cross-validation instead of 5 for faster performance\n",
    "    scoring='accuracy',\n",
    "    n_jobs=-1,  # Use all available cores\n",
    "    random_state=42\n",
    ")\n",
    "\n",
    "# Fit the randomized search\n",
    "random_search.fit(X_train, y_train)\n",
    "\n",
    "# Print the best parameters and score\n",
    "print(\"Best parameters:\", random_search.best_params_)\n",
    "print(\"Best score:\", random_search.best_score_)\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import (\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    roc_auc_score,\n",
    "    roc_curve,\n",
    ")\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create Random Forest model\n",
    "rf_model = RandomForestClassifier(\n",
    "    n_estimators=100,  # Number of trees\n",
    "    class_weight='balanced',\n",
    "    random_state=1,\n",
    "    min_samples_split= 2,\n",
    "    min_samples_leaf= 1,\n",
    "    max_features='sqrt',\n",
    "    max_depth=None\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "rf_model.fit(X_train_resampled, y_train_resampled)\n",
    "\n",
    "# Make predictions\n",
    "y_pred = rf_model.predict(X_test_scaled)\n",
    "y_pred_prob = rf_model.predict_proba(X_test_scaled)[:, 1]\n",
    "\n",
    "# Print model evaluation metrics\n",
    "print(\"Classification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "\n",
    "# Compute AUC-ROC score\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "# Plot ROC curve\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f\"AUC-ROC = {auc_roc:.4f}\", color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.xlabel(\"False Positive Rate (FPR)\")\n",
    "plt.ylabel(\"True Positive Rate (TPR)\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.grid()\n",
    "plt.show()"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The hyperparameters tuning gave us some improvement, but previous models were just better."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Neural network"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "# Import necessary libraries\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, roc_curve\n",
    "\n",
    "# Define the model\n",
    "model = keras.Sequential([\n",
    "    keras.layers.Dense(64, activation='relu', input_shape=(X_train_resampled.shape[1],)),\n",
    "    keras.layers.Dense(32, activation='relu'),\n",
    "    keras.layers.Dense(1, activation='sigmoid')  # Output layer for binary classification\n",
    "])\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()])\n",
    "\n",
    "# Train the model\n",
    "history = model.fit(X_train_resampled, y_train_resampled,\n",
    "                   epochs=17,\n",
    "                   batch_size=64,\n",
    "                   validation_split=0.2)\n",
    "\n",
    "# Evaluate the model\n",
    "loss, accuracy, precision, recall = model.evaluate(X_test_scaled, y_test)\n",
    "print(f\"Loss: {loss}\")\n",
    "print(f\"Accuracy: {accuracy}\")\n",
    "print(f\"Precision: {precision}\")\n",
    "print(f\"Recall: {recall}\")\n",
    "\n",
    "# Make predictions\n",
    "y_pred_prob = model.predict(X_test_scaled)\n",
    "y_pred = (y_pred_prob > 0.5).astype(int)  # Convert probabilities to class labels\n",
    "\n",
    "# Create figure for all plots\n",
    "plt.figure(figsize=(15, 10))\n",
    "\n",
    "# 1. Classification Report and Confusion Matrix\n",
    "print(\"\\nClassification Report:\")\n",
    "class_report = classification_report(y_test, y_pred)\n",
    "print(class_report)\n",
    "\n",
    "# Plot confusion matrix with seaborn\n",
    "plt.subplot(2, 2, 1)\n",
    "conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "\n",
    "# 2. ROC Curve\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "\n",
    "plt.subplot(2, 2, 2)\n",
    "plt.plot(fpr, tpr, label=f'AUC-ROC = {auc_roc:.4f}', color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title('ROC Curve')\n",
    "plt.xlabel('False Positive Rate (FPR)')\n",
    "plt.ylabel('True Positive Rate (TPR)')\n",
    "plt.legend(loc='lower right')\n",
    "plt.grid(True)\n",
    "\n",
    "# 3. Training History\n",
    "plt.subplot(2, 2, 3)\n",
    "plt.plot(history.history['loss'], label='Training Loss')\n",
    "plt.plot(history.history['val_loss'], label='Validation Loss')\n",
    "plt.title('Model Loss During Training')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(2, 2, 4)\n",
    "plt.plot(history.history['accuracy'], label='Training Accuracy')\n",
    "plt.plot(history.history['val_accuracy'], label='Validation Accuracy')\n",
    "plt.title('Model Accuracy During Training')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "\n",
    "# Save the visualizations\n",
    "def save_evaluation_results(y_test, y_pred, y_pred_prob, history, model_name=\"neural_network\"):\n",
    "    \"\"\"\n",
    "    Save evaluation results to files\n",
    "    \"\"\"\n",
    "    # Create classification report\n",
    "    report = classification_report(y_test, y_pred)\n",
    "    with open(f'{model_name}_classification_report.txt', 'w') as f:\n",
    "        f.write(report)\n",
    "\n",
    "    # Save confusion matrix visualization\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    sns.heatmap(confusion_matrix(y_test, y_pred), annot=True, fmt='d', cmap='Blues')\n",
    "    plt.title('Confusion Matrix')\n",
    "    plt.xlabel('Predicted')\n",
    "    plt.ylabel('Actual')\n",
    "    plt.savefig(f'{model_name}_confusion_matrix.png')\n",
    "    plt.close()\n",
    "\n",
    "    # Save ROC curve\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "    fpr, tpr, _ = roc_curve(y_test, y_pred_prob)\n",
    "    plt.plot(fpr, tpr, label=f'AUC-ROC = {auc_roc:.4f}')\n",
    "    plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "    plt.title('ROC Curve')\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.legend()\n",
    "    plt.grid(True)\n",
    "    plt.savefig(f'{model_name}_roc_curve.png')\n",
    "    plt.close()\n",
    "\n",
    "# Call the save function\n",
    "save_evaluation_results(y_test, y_pred, y_pred_prob, history)"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "After first training, the results seems quite good, but let's do some hyperparameters tuning."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from itertools import product\n",
    "from tensorflow import keras\n",
    "\n",
    "# Define hyperparameter grid\n",
    "learning_rates = [0.01, 0.001, 0.0001]\n",
    "neurons = [32, 64, 128]\n",
    "batch_sizes = [32, 64, 128]\n",
    "\n",
    "best_accuracy = 0\n",
    "best_params = None\n",
    "\n",
    "# Iterate over combinations of hyperparameters\n",
    "for lr, n, batch in product(learning_rates, neurons, batch_sizes):\n",
    "    model = keras.Sequential([\n",
    "        keras.layers.Input(shape=(X_train_resampled.shape[1],)),  # Explicit Input layer\n",
    "        keras.layers.Dense(n, activation='relu'),\n",
    "        keras.layers.Dense(n // 2, activation='relu'),\n",
    "        keras.layers.Dense(1, activation='sigmoid')\n",
    "    ])\n",
    "\n",
    "    model.compile(\n",
    "        optimizer=keras.optimizers.Adam(learning_rate=lr),\n",
    "        loss='binary_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "\n",
    "    history = model.fit(X_train_resampled, y_train_resampled, epochs=10, batch_size=batch, validation_split=0.2, verbose=0)\n",
    "    val_accuracy = max(history.history['val_accuracy'])\n",
    "\n",
    "    if val_accuracy > best_accuracy:\n",
    "        best_accuracy = val_accuracy\n",
    "        best_params = (lr, n, batch)\n",
    "\n",
    "print(f\"Best parameters: Learning rate={best_params[0]}, Neurons={best_params[1]}, Batch size={best_params[2]}\")"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, roc_curve\n",
    "\n",
    "# Define the model with best parameters\n",
    "final_model = keras.Sequential([\n",
    "    keras.layers.Dense(32, activation='relu', input_shape=(X_train_resampled.shape[1],)),\n",
    "    keras.layers.Dense(16, activation='relu'),  # Half the neurons as per your original structure\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "# Compile the model with best learning rate\n",
    "final_model.compile(\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=0.01),\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['accuracy', tf.keras.metrics.Precision(), tf.keras.metrics.Recall()]\n",
    ")\n",
    "\n",
    "# Train the model with best batch size\n",
    "history = final_model.fit(\n",
    "    X_train_resampled,\n",
    "    y_train_resampled,\n",
    "    epochs=17,\n",
    "    batch_size=32,  # Best batch size from grid search\n",
    "    validation_split=0.2\n",
    ")\n",
    "\n",
    "# Evaluate the final model with additional metrics and visualizations\n",
    "y_pred_prob = final_model.predict(X_test_scaled)\n",
    "\n",
    "# Instead of 0.5, try a higher threshold like 0.7 or 0.8\n",
    "y_pred = (y_pred_prob > 0.65).astype(int)\n",
    "\n",
    "#y_pred = (y_pred_prob > 0.5).astype(int)\n",
    "\n",
    "# Classification Report\n",
    "print(\"\\nClassification Report:\")\n",
    "class_report = classification_report(y_test, y_pred)\n",
    "print(class_report)\n",
    "\n",
    "# Confusion Matrix\n",
    "conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(conf_matrix)\n",
    "\n",
    "# Plot Confusion Matrix\n",
    "plt.figure(figsize=(8, 6))\n",
    "sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "plt.show()\n",
    "\n",
    "# AUC-ROC Score and Curve\n",
    "auc_roc = roc_auc_score(y_test, y_pred_prob)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob)\n",
    "\n",
    "print(f\"\\nAUC-ROC Score: {auc_roc:.4f}\")\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr, tpr, label=f'AUC-ROC = {auc_roc:.4f}', color='blue')\n",
    "plt.plot([0, 1], [0, 1], linestyle='--', color='gray')\n",
    "plt.title('ROC Curve')\n",
    "plt.xlabel('False Positive Rate (FPR)')\n",
    "plt.ylabel('True Positive Rate (TPR)')\n",
    "plt.legend(loc='lower right')\n",
    "plt.grid(True)\n",
    "plt.show()\n",
    "\n",
    "# Optional: Save the final model for future use\n",
    "final_model.save('final_model.h5')"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Actually, the results are bad. We will try to do some work to overcome this issue.\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from tensorflow import keras\n",
    "from sklearn.metrics import precision_recall_curve, classification_report, confusion_matrix, roc_auc_score\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "# Adjusted Model Architecture\n",
    "final_model = keras.Sequential([\n",
    "    keras.layers.Dense(128, activation='relu',\n",
    "                       input_shape=(X_train_resampled.shape[1],),\n",
    "                       kernel_regularizer=keras.regularizers.l2(0.005)),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dropout(0.1),\n",
    "\n",
    "    keras.layers.Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l2(0.005)),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dropout(0.1),\n",
    "\n",
    "    keras.layers.Dense(32, activation='relu', kernel_regularizer=keras.regularizers.l2(0.005)),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.Dropout(0.1),\n",
    "\n",
    "    keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "# Adjusted Learning Rate\n",
    "final_model.compile(\n",
    "    optimizer=keras.optimizers.Adam(learning_rate=0.01),  # Reduced initial learning rate\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['accuracy', keras.metrics.AUC(name='auc')]\n",
    ")\n",
    "\n",
    "# Callbacks\n",
    "early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', patience=6, restore_best_weights=True)\n",
    "reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, min_lr=0.0001)\n",
    "\n",
    "# Train Model\n",
    "history = final_model.fit(\n",
    "    X_train_resampled,\n",
    "    y_train_resampled,\n",
    "    epochs=20,\n",
    "    batch_size=32,  # Smaller batch size\n",
    "    validation_split=0.2,\n",
    "    callbacks=[early_stopping, reduce_lr],\n",
    "    class_weight={0: 2.0, 1: 1.0}  # Adjusted class weights\n",
    ")\n",
    "\n",
    "# Predictions and Threshold Adjustment\n",
    "y_pred_prob = final_model.predict(X_test_scaled)\n",
    "precision, recall, thresholds = precision_recall_curve(y_test, y_pred_prob)\n",
    "f1_scores = 2 * (precision * recall) / (precision + recall)\n",
    "best_threshold = thresholds[np.argmax(f1_scores)]\n",
    "print(f\"Best Threshold: {best_threshold}\")\n",
    "\n",
    "# Apply Best Threshold\n",
    "y_pred = (y_pred_prob > best_threshold).astype(int)\n",
    "\n",
    "# Evaluate Model\n",
    "print(\"\\nClassification Report:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    "conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "print(\"\\nConfusion Matrix:\")\n",
    "print(conf_matrix)\n",
    "\n",
    "# Confusion Matrix Plot\n",
    "plt.figure(figsize=(8, 6))\n",
    "sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues')\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "plt.show()\n",
    "\n",
    "# Training History Plots\n",
    "plt.figure(figsize=(12, 4))\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.plot(history.history['loss'], label='Training Loss')\n",
    "plt.plot(history.history['val_loss'], label='Validation Loss')\n",
    "plt.title('Model Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(history.history['accuracy'], label='Training Accuracy')\n",
    "plt.plot(history.history['val_accuracy'], label='Validation Accuracy')\n",
    "plt.title('Model Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We got some improvements, but in the end other algorithms worked better."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Summary"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "- The dataset was highly imbalanced (99.52% approvals), requiring resampling techniques for fair model training.\n",
    "- Logistic Regression and SVM achieved nearly perfect classification, with AUC-ROC scores close to 1.0.\n",
    "- Decision Trees and Random Forests performed well but showed sensitivity to hyperparameters.\n",
    "- Neural Networks, despite extensive tuning, struggled to outperform traditional models, suggesting simpler models might be more effective for this task.\n",
    "- Hyperparameter tuning improved performance across models, particularly in Random Forests and Decision Trees."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Significant Conclusions and Implications\n",
    "- Simple models (Logistic Regression, SVM) are highly effective, achieving high precision and recall with minimal computational cost.\n",
    "- Data preprocessing significantly impacts model accuracy, especially in handling imbalanced data and feature scaling.\n",
    "- Neural Networks may not always be the best choice, particularly when structured tabular data is involved.\n",
    "- Further research is needed to generalize findings to different datasets and real-world financial scenarios."
   ],
   "metadata": {}
  }
 ]
}