Algorithmic bias
Also called AI bias, machine learning bias
Algorithmic bias is when an automated model produces systematically worse outcomes for certain groups of people, such as by sex, ethnicity, age or postcode. It usually comes from skewed training data, proxy variables or design choices. In financial services it can breach equality, consumer credit and data protection law.
How it works
Bias often enters through history. A credit model trained on past lending decisions can learn the patterns of earlier discrimination. Variables such as postcode or shopping data can act as proxies for protected characteristics even when those are excluded. Firms test models by comparing approval rates, error rates and pricing across groups before deployment and again in production.
The legal risk is concrete. Discriminatory outcomes can breach the UK Equality Act 2010, the US Equal Credit Opportunity Act and the GDPR fairness principle. The EU AI Act requires providers of high-risk systems, including credit scoring, to examine training data for possible biases. Those Annex III duties apply from 2 December 2027.
The rules
Requires training, validation and testing data to be examined for possible biases and for measures to detect, prevent and mitigate them.
Providers of Annex III high-risk AI systems, including creditworthiness assessmentProhibits direct and indirect discrimination on protected characteristics in services, employment and public functions.
Service providers, employers and public bodies in Great BritainProhibits credit discrimination on protected bases and requires specific reasons when credit is denied.
Creditors in the United StatesCommon questions
What causes algorithmic bias?
The main causes are unrepresentative or historical training data, proxy variables that correlate with protected characteristics and design choices such as the target a model optimises. Bias can also appear after deployment when the customer population or economic conditions change.
Can removing protected characteristics stop bias?
Not on its own. Models can infer sex, ethnicity or age from other data, such as postcode, spending patterns or device type. Firms usually keep protected data in a separate test environment so they can measure outcome gaps across groups.
How do firms measure algorithmic bias?
Common tests compare approval rates, error rates and prices across groups. Measures include the ratio of approval rates between groups and differences in false positive or false negative rates. The right metric depends on the decision and the law that applies.
