Industry-Grade SMOTE Rebalancing

What is SMOTE?
SMOTE stands for Synthetic Minority Over-Sampling Technique. It’s a clever way to help computer programs learn fairly when one group in the data is really small compared to another.
Imagine you have 100 red apples and only 5 green apples. A computer might never learn about green apples properly. SMOTE creates “new” green apples in a smart way so the computer gets a fair chance.

1. Why SMOTE matters

  • In many datasets, one group (the “majority”) has tons of examples and another group (the “minority”) has very few.

  • If a computer program only sees the big group, it might ignore the small one. That leads to unfair or weak results.

  • SMOTE helps by creating extra examples for the small group so the program can learn about it properly.

2. How SMOTE works (simple steps)

Here’s how it happens:

  1. Pick a random example from the minority group (e.g., one green apple).

  2. Find similar examples from the same group (its “nearest neighbours”).

  3. Create a new example somewhere in between the picked example and a neighbour (like making a slightly different green apple).

  4. Repeat until the minority group has enough examples and the data is more balanced.

3. What “Industry-Grade SMOTE” means in BiasClean

In our system, we don’t just randomly create data. We use SMOTE smartly for real business and research use:

  • Very low data loss: We only remove up to a small percent of the majority group so we don’t throw away too much real information.

  • Controlled creation: Any new examples stay true to the minority group—they don’t mix groups or change protected features (like gender or ethnicity).

  • Evidence-based: We only rebalance when the bias is strong and the feature (like gender or region) counts a lot in that domain.

  • Ready for real use: After rebalancing, we check results, show easy-to-read reports and make sure the data is ready for production.

4. Step-by-step in BiasClean

Step A – Minimal Undersampling

We slightly reduce the majority group (e.g., from red apples) only if there’s clear evidence they dominate unfairly. We cap removal (for example, no more than 8%) so we keep most real data.

Step B – SMOTE-style Oversampling

For the minority group (e.g., green apples), we add synthetic examples:

  • We pick real examples from that group.

  • We generate new ones that are similar but not exactly the same.

  • We keep protected attributes unchanged (so, still green apples, same size, etc.).

  • We avoid creating weird or unrealistic data.

Step C – Domain-aware focus

Because we have different “domains” (Justice, Health, Finance, etc.), we look at which features (like age, gender, region) matter most in each domain. The system applies SMOTE where it will make the biggest difference based on real UK data.

Step D – Validation & Metrics

After rebalancing:

  • We compare bias before and after.

  • We show how much data was changed or added.

  • We give a clear pass/fail for “Ready for production”.

  • We include simple visuals: how groups are distributed now, how many examples were added/removed.

5. What you see as a user

When you upload your dataset and select “Industry mode”:

  • The system detects which features are unfair.

  • It shows you how much they’re unfair (e.g., Group A has only half as many examples as expected).

  • It runs the SMOTE-based rebalancing.

  • You download the new balanced dataset.

  • You get a clear report: what changed, why it changed, and whether it’s ready for real-world use.

6. Real-world example

Imagine you’re working with a hiring dataset: 95 men and 5 women. The “women” group is very small.

Step A: Maybe remove 3 men (so data loss stays under 8%).

Step B: Create synthetic “woman” records until you have a more balanced mix (say 92 men and 25 women).

Step C: Since gender is very important in hiring, the system focuses on that.

Step D: You get a new dataset + a report saying “Gender bias significantly reduced, 97% of original data retained”.

7. Simple summary

SMOTE is like giving the smaller group a helping hand by creating extra, realistic examples. And “industry-grade” means we do this carefully, scientifically and as part of a full production system—not just for fun.