TABULAR MACHINE LEARNING / ACADEMIC PROJECT
Almost perfect.
Look closer.
When 99.52% of records share one label, a high accuracy score is only the beginning. This credit-card classification study compares model families and examines the errors that an average can hide.
View code on GitHub- Source records
- 25,128
- Majority label
- 99.52%
- Test records
- 10,052
- Minority test records
- 48
01 / THE QUESTION
What does a good
score leave out?
The notebook studies a binary Status label in credit-card application data. It assumes 1 means approved and 0 means rejected. The dataset documentation was not supplied, so this article uses the original Status labels rather than treating them as verified lending outcomes.
There are 25,007 Status 1 records and only 121 Status 0 records. After a stratified 60/40 split, the test set contains 10,004 Status 1 records and 48 Status 0 records. Predicting Status 1 for everyone would score 99.52% accuracy while identifying none of the minority records.
02 / LOOK INSIDE THE SCORE
Same test set.
Different mistakes.
Switch between saved evaluations and the calculated majority baseline. Each dot below represents one of the 48 actual Status 0 records.
Each row has its own denominator, so both classes are visible.
Dark: correctly predicted 0 · Pale: predicted 1
Status 0 recall: 50.0%
Status 0 F1: 0.658
100 trees; unrestricted depth. Detects only half of the 48 Status 0 records.
The revised random forest scores 99.75% accuracy but identifies only 24 of the 48 Status 0 records. The initial forest identifies 28. Its revision reduces errors in the majority class while worsening minority recall. Whether that tradeoff is acceptable requires an explicit objective; accuracy cannot decide it.
View all matrices and derived metrics
| Model | Accuracy | Status 0 recall | Status 0 F1 | ROC AUC | Matrix [00, 01; 10, 11] |
|---|---|---|---|---|---|
| Always Status 1 | 99.522% | 0.0% | 0.000 | Not reported | 0, 48, 0, 10004 |
| Logistic regression | 99.960% | 100.0% | 0.960 | 1.0000 | 48, 0, 4, 10000 |
| Linear SVM | 99.980% | 100.0% | 0.980 | 1.0000 | 48, 0, 2, 10002 |
| Decision tree · initial | 99.851% | 77.1% | 0.831 | 0.8957 | 37, 11, 4, 10000 |
| Decision tree · revised | 99.910% | 91.7% | 0.907 | 0.9582 | 44, 4, 5, 9999 |
| Random forest · initial | 99.652% | 58.3% | 0.615 | 0.9965 | 28, 20, 15, 9989 |
| Random forest · revised | 99.751% | 50.0% | 0.658 | 0.9994 | 24, 24, 1, 10003 |
| Neural network · fixed threshold | 99.801% | 77.1% | 0.787 | 0.9775 | 37, 11, 9, 9995 |
| Neural network · test-tuned | 99.901% | 81.3% | 0.886 | Not reported | 39, 9, 1, 10003 |
03 / THE WORKFLOW
Rebalance training.
Keep the test skew.
- Prepare the features. One-hot encode categorical columns, remove the applicant identifier and job title, and standardize five numeric columns using training-set statistics.
- Resample the training partition. SMOTETomek uses a 0.4 minority-to-majority sampling ratio. The saved output contains 21,004 training records after resampling: 15,003 Status 1 and 6,001 Status 0. The test distribution remains unchanged.
- Compare model families. Fit logistic regression, linear SVM, decision trees, random forests, and dense neural networks. Explore regularization, tree depth, class weights, and network architecture.
- Inspect the minority class. Read confusion matrices alongside ROC AUC and class-specific scores. The two-decimal classification reports round several different accuracies to 1.00; the explorer calculates them directly from the saved counts.
04 / THE EVALUATION AUDIT
Strong results deserve
stronger checks.
Logistic regression and the linear SVM identify all 48 minority records in the saved test output, with four and two majority-class errors respectively. Their ROC AUC is reported as 1.0000, rounded to four decimals. These are recorded results, not proof of performance on future applications.
Feature timing needs verification. Total_Bad_Debt and Total_Good_Debt remain in the predictors. Without documentation of how Status was created and when these fields were measured, target leakage is a possibility that needs investigation, not a confirmed finding.
Validation needs separation. The SVM and decision-tree searches run cross-validation after the full training partition has already been resampled. Scaling and resampling should instead be fitted within each fold. Neural networks use validation_split after resampling, and their saved logs show zero validation AUC or precision/recall in several runs. The label composition and ordering of that validation slice need checking.
The final threshold uses test labels. The last neural network chooses its threshold from the test-set precision-recall curve, optimizing F1 for Status 1. Its displayed outcome is exploratory and cannot serve as an independent final test. A separate validation set is needed for threshold selection.
05 / WHAT COMES NEXT
Make the experiment
harder to fool.
First, establish the data source, target definition, and feature availability at prediction time. Then rerun with preprocessing and resampling inside a cross-validation pipeline, explicit stratified validation, and an untouched final test. Synthetic interpolation across one-hot categories also deserves scrutiny.
Report both classes, precision-recall curves for Status 0, calibration, and uncertainty across splits. Only 48 minority test records means every additional error changes minority recall by about 2.1 percentage points.
This is a retrospective classification exercise, not a system for making lending decisions. The notebook does not establish fairness, prospective validity, or the consequences of either error type.