Technical Implementation
BiasClean v2.0 has been designed to be simple for users and powerful behind the scenes. This page explains, in clear everyday language, how the system works inside. You do not need technical knowledge — the aim is to help anyone understand what happens when they upload a dataset.
1. What Happens When You Upload a File
When you upload a CSV file, the system goes through four main steps:
Reads the data
It checks that the file is valid and that the columns are readable.Understands the “domain” you selected
For example: Justice, Health, Finance, Education, Hiring, Business or Governance.Finds the seven fairness features in your data
(Ethnicity, SES, Region, Age, Gender, Disability Status, Migration Status)Runs the statistical checks and fairness detection
All of this happens automatically in a few seconds.
2. The Seven Fairness Features
The system is built around seven demographic features that are widely used in the UK to measure fairness:
Ethnicity
Socioeconomic Status (SES)
Region
Age
Gender
Disability Status
Migration Status
If some features are missing in your dataset, the system still works — it simply evaluates the features that are present.
3. Domain-Specific Weighting
Each domain has its own “weight matrix”.
This helps the system understand which features matter most in each area.
Examples:
In Health, Disability and Gender matter more.
In Finance, SES and Region matter more.
In Justice, Ethnicity and SES matter more.
These weights come from official UK evidence, ensuring fairness decisions match real-world conditions.
4. How the Detection Engine Works
The detection engine looks for imbalances using simple maths:
It counts how many people belong to each demographic group.
It compares this with UK population patterns (e.g., Census data).
It scores each group as:
Green (good), Amber (medium) or Red (serious).
The system then combines these checks using the domain weights.
This gives you a final “fairness score”.
5. Rebalancing With Industry-Grade SMOTE
If you enable Industry Mode, the system uses controlled SMOTE-style balancing:
No more than a small amount of data is removed (capped).
New synthetic examples are added only for under-represented groups.
Protected attributes (Ethnicity, Gender, etc.) are never changed.
Only safe numerical features get small, controlled variations.
This keeps the dataset realistic, fair and ready for AI models.
6. Quality and Safety Checks
BiasClean performs several internal safety checks:
Data Validity Check: Ensures your file is readable.
Missing Column Check: Alerts you if key features are missing.
Distribution Check: Identifies extreme imbalances.
Post-SMOTE Check: Ensures new data remains realistic and consistent.
Data Retention Check: Shows how much real data remains (usually above 90%).
These checks protect you from incorrect or misleading outputs.
7. What the System Returns to You
After processing, BiasClean sends back:
A. Rebalanced Dataset (CSV)
A corrected and fairer version of your original file.
B. Full Analytics Report (PDF/HTML)
Shows how fair the dataset was before and after.
Includes simple explanations and charts.
C. Summary Sheet
Useful for sharing with students, teachers, colleagues or regulators.
D. Visual Charts
Before/after plots, fairness improvements and group distributions.
Everything is written in plain language.
8. Built With Modern, Reliable Tools
BiasClean v2.0 is created using trusted, industry-standard libraries:
Python 3.11
NumPy and Pandas for data handling
SciPy for statistics
Matplotlib for charts
Custom SMOTE engine carefully designed for fairness use-cases
Domain-specific weight matrices based on UK evidence
All code is transparent and available in the public GitHub, https://github.com/AI-Fairness-com/BiasClean, repository.
9. Why Technical Implementation Matters
Understanding how the system works helps users:
Trust the fairness results
Explain the process to others
Share the report with confidence
Use the dataset in AI models, research or business
Meet compliance and ethical standards