A mail-order company has applications for its logistics centre pre-sorted by a model trained on the hiring decisions of the past ten years. Because mostly men were hired during that time, the model has learned to downgrade features that appear more often in women’s CVs, such as part-time periods. Nobody entered sex itself as a feature. A model takes over the patterns in its data, including the unfair ones, and applies them with great consistency.
ISO/IEC 42001 deals with the topic in two places: in Annex C among the risk sources related to data and machine learning, and in control A.5.4, which requires the impact on individuals and groups to be assessed. The standard also lists fairness as an objective of its own, because automated decisions can treat certain people or groups unfairly.
What is behind it?
Bias means that a system systematically treats people or groups differently in a way that cannot be justified on objective grounds. Bias usually creeps in unnoticed: from the data, from decisions during development and from the conditions of use. It needs no ill intent.
Sources of bias
- Historical bias. The data reflects past decisions or social inequality, and the model learns them as a rule.
- Under-representation. Some groups appear too rarely in the data. For them, the model works less accurately.
- Biased targets and labels. The target measures something other than intended, such as past treatment costs as a stand-in for actual treatment need. Or the labels carry the prejudices of the people who assigned them.
- Proxy features. Postcode, first name or gaps in a CV correlate statistically with protected characteristics and carry them into the model.
- Feedback. The system’s decisions produce tomorrow’s data and amplify an existing imbalance.
- Bias in deployment. The system is applied to a group it was never built or tested for.
Fairness is a decision
There are several measures of fairness, such as equal selection rates or equal error rates per group. Research has shown that such measures generally cannot all be met at the same time. The organization therefore has to decide which yardstick fits the use and justify that choice.
Impact
Fundamental rights are at stake, above all the right to equal treatment, often in access to work, credit, education or services. For an individual, a biased decision can cost a job or a home. For the organization, lawsuits, fines and severe reputational damage loom. A biased model discriminates at scale and under the appearance of objectivity.
Practical examples
Creditworthiness by place of residence. A bank’s scoring model uses the postcode. Residents of certain districts receive worse scores, although their financial situation matches that of other applicants. Because an above-average share of people with a migration background live in these districts, the postcode acts like a marker of origin.
Face recognition with unequal error rates. An access system at a plant recognizes employees with darker skin less reliably. They have to go to the manual check far more often and lose time every morning. The system’s overall accuracy looked excellent in testing, because it was never broken down by group.
Urgency in customer service. A language model sorts complaints by urgency. It rates messages written in plain language or by people whose first language is not the local one as less urgent more often. Precisely these customers wait longer for an answer.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.5 Assessing impacts of AI systems: The organization assesses the impact on individuals and groups, including fairness and human rights, and pays attention to particularly vulnerable groups such as children, older people, people with disabilities and workers. At the level of society, it asks whether the system can reinforce entrenched prejudices.
- A.6.1 Management guidance for AI system development: Fairness is set as an objective of responsible development and built into the development life cycle.
- A.7 Data for AI systems: At acquisition, characteristics of the people concerned and known or potential biases are documented. The data quality requirements take into account how bias affects performance and fairness.
Detection:
- A.6.2 AI system life cycle: Verification and validation also assess the system for impacts on individuals and groups, with test data that reflects the domain of use. Monitoring in operation continues this check.
Response:
- A.9 Use of AI systems: Fairness is one of the objectives of responsible use. Human reviewers may override the system’s decisions, and the system is used only within its intended use.
- A.6.2 AI system life cycle: Identified bias is removed through adjustments to data or model and a tested update.
Link to the AI Act
Under Article 10 of the AI Act (Regulation (EU) 2024/1689), providers of high-risk systems examine their data for possible biases that could affect health, safety or fundamental rights or lead to prohibited discrimination, and they take measures to detect, prevent and mitigate them. Article 15(4) requires safeguards against feedback of biased outputs in systems that keep learning. Many areas of use in Annex III, such as employment, education and creditworthiness, count as high-risk precisely because of the risk of discrimination. Public bodies and private providers of public services that deploy high-risk systems from Annex III, as well as deployers of systems for credit scoring and for pricing in life and health insurance, carry out a fundamental rights impact assessment under Article 27 before use. In Germany, the General Equal Treatment Act (AGG) additionally prohibits discrimination, for example in working life (→ EU AI Act).
Related threats
- T04 Poor data quality: Unbalanced and faulty data is the most common cause of bias.
- T02 Lack of transparency and explainability: In opaque systems, discrimination stays hidden for a long time.
- MU-06 Use on untested groups: The misuse pattern describes bias in deployment from the angle of usage rules.
Sources
- ISO/IEC 42001:2023, Annex C.2.5 and C.3.4, control A.5.4 with Annex B.5.4 and B.7.4
- Regulation (EU) 2024/1689 (AI Act), Articles 10, 15 and 27 and Annex III
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems