In production systems, the events you care about are almost always the rare ones. Fraud in a payment stream. A failing node in a storage cluster. A positive case in a medical screening dataset. I have dealt with this from the engineering side for years, and when I started studying machine learning seriously, I wanted to understand how the modeling side handles it.
So I built a small benchmark and ran it properly. This post is a summary of what I found. The code and full results are public at github.com/JonMirzoDev/imbalanced-benchmarks.
The problem with accuracy
Take a medical dataset where 2.3% of samples are positive. A model that predicts “negative” for every single sample scores 97.7% accuracy. It also catches zero positive cases, which makes it useless.
This is not a corner case. It is the normal situation in fraud detection, disease screening, anomaly detection, and most other problems where a classifier would actually be useful. If you evaluate with accuracy, you can ship a model that looks great and does nothing.
The benchmark
I compared seven method configurations on three medical datasets with different levels of imbalance:
| Dataset | Positive class |
|---|---|
| mammography | ~2.3% (severe) |
| thyroid_sick | ~6% (moderate) |
| pima diabetes | ~35% (mild) |
The methods: logistic regression and random forest, each untreated (baseline), with cost-sensitive class weighting, and with SMOTE oversampling. For the forest I also added random undersampling. Every configuration ran under 5-fold stratified cross-validation with a fixed seed, scored with PR-AUC, ROC-AUC, F1, balanced accuracy, and recall.
One implementation detail matters more than any other, so I will show it:
from imblearn.pipeline import Pipeline
from imblearn.over_sampling import SMOTE
pipeline = Pipeline([
("scale", StandardScaler()),
("smote", SMOTE(random_state=42)),
("clf", LogisticRegression(max_iter=2000)),
])
SMOTE lives inside the pipeline. That means during cross-validation it only ever sees the training folds. If you apply SMOTE to the whole dataset before splitting, synthetic samples generated from your training data leak into your test set, and your scores come out better than they should. This is the most common mistake in imbalanced-learning code, and it is silent.
What the numbers said
Here are the mammography results (the severely imbalanced dataset), sorted by PR-AUC:
| Method | PR-AUC | Recall | F1 |
|---|---|---|---|
| forest_baseline | 0.755 | 0.546 | 0.665 |
| forest_weighted | 0.736 | 0.635 | 0.678 |
| forest_smote | 0.711 | 0.754 | 0.635 |
| forest_undersample | 0.643 | 0.885 | 0.329 |
| logreg_baseline | 0.618 | 0.415 | 0.539 |
| logreg_weighted | 0.547 | 0.846 | 0.281 |
| logreg_smote | 0.541 | 0.850 | 0.284 |
Two things surprised me.
First: ranked by PR-AUC, the untreated baselines win. I expected the imbalance treatments to improve the headline metric. They did not. What they actually did was trade ranking quality for recall. Class weighting lifted logistic regression’s recall from 0.415 to 0.846, which means it catches twice as many positive cases, but precision paid for it and PR-AUC went down.
Second: undersampling is the bluntest instrument. It reached the highest recall on the forest (0.885) and destroyed everything else (F1 dropped from 0.665 to 0.329). Throwing away most of your majority class is a very expensive way to move the decision threshold.
On the mildly imbalanced diabetes dataset, all methods landed within noise of each other. The treatments only matter when the imbalance is severe.
The takeaway I keep
The imbalance treatments are not “fixes” that make a model better. They are ways of moving along the precision-recall trade-off, and whether that trade is good depends entirely on the cost of a missed positive versus a false alarm. In a medical screening setting, recall of 0.55 might be unacceptable and recall of 0.85 with more false alarms might be fine. In another setting the opposite is true.
So the real work is not choosing SMOTE versus class weights. It is deciding what errors cost, choosing metrics that reflect that, and making sure your evaluation is not leaking. That conclusion sounds obvious written down. Running the benchmark is what made it concrete for me, and I later found the same pattern reported in published work on multiclass driver distraction, where SMOTE made almost no difference once the sample counts per class were large enough.
If you want to check my numbers, the repository has everything: datasets are fetched automatically, the seed is fixed, and python -m benchmark.run reproduces the full results table on a laptop in a few minutes.