Zum Hauptinhalt springen
AI threat · ISO/IEC 42001 Annex C

T12 Bias and discrimination

Updated on 6 min Reviewed by: Cenedril Editorial
ISO/IEC 42001EU AI Act

A mail-order company has applications for its logistics centre pre-sorted by a model trained on the hiring decisions of the past ten years. Because mostly men were hired during that time, the model has learned to downgrade features that appear more often in women’s CVs, such as part-time periods. Nobody entered sex itself as a feature. A model takes over the patterns in its data, including the unfair ones, and applies them with great consistency.

ISO/IEC 42001 deals with the topic in two places: in Annex C among the risk sources related to data and machine learning, and in control A.5.4, which requires the impact on individuals and groups to be assessed. The standard also lists fairness as an objective of its own, because automated decisions can treat certain people or groups unfairly.

What is behind it?

Bias means that a system systematically treats people or groups differently in a way that cannot be justified on objective grounds. Bias usually creeps in unnoticed: from the data, from decisions during development and from the conditions of use. It needs no ill intent.

Sources of bias

  • Historical bias. The data reflects past decisions or social inequality, and the model learns them as a rule.
  • Under-representation. Some groups appear too rarely in the data. For them, the model works less accurately.
  • Biased targets and labels. The target measures something other than intended, such as past treatment costs as a stand-in for actual treatment need. Or the labels carry the prejudices of the people who assigned them.
  • Proxy features. Postcode, first name or gaps in a CV correlate statistically with protected characteristics and carry them into the model.
  • Feedback. The system’s decisions produce tomorrow’s data and amplify an existing imbalance.
  • Bias in deployment. The system is applied to a group it was never built or tested for.

Fairness is a decision

There are several measures of fairness, such as equal selection rates or equal error rates per group. Research has shown that such measures generally cannot all be met at the same time. The organization therefore has to decide which yardstick fits the use and justify that choice.

Impact

Fundamental rights are at stake, above all the right to equal treatment, often in access to work, credit, education or services. For an individual, a biased decision can cost a job or a home. For the organization, lawsuits, fines and severe reputational damage loom. A biased model discriminates at scale and under the appearance of objectivity.

Practical examples

Creditworthiness by place of residence. A bank’s scoring model uses the postcode. Residents of certain districts receive worse scores, although their financial situation matches that of other applicants. Because an above-average share of people with a migration background live in these districts, the postcode acts like a marker of origin.

Face recognition with unequal error rates. An access system at a plant recognizes employees with darker skin less reliably. They have to go to the manual check far more often and lose time every morning. The system’s overall accuracy looked excellent in testing, because it was never broken down by group.

Urgency in customer service. A language model sorts complaints by urgency. It rates messages written in plain language or by people whose first language is not the local one as less urgent more often. Precisely these customers wait longer for an answer.

Effective controls

The following ISO/IEC 42001 controls counter this threat.

Prevention:

  • A.5 Assessing impacts of AI systems: The organization assesses the impact on individuals and groups, including fairness and human rights, and pays attention to particularly vulnerable groups such as children, older people, people with disabilities and workers. At the level of society, it asks whether the system can reinforce entrenched prejudices.
  • A.6.1 Management guidance for AI system development: Fairness is set as an objective of responsible development and built into the development life cycle.
  • A.7 Data for AI systems: At acquisition, characteristics of the people concerned and known or potential biases are documented. The data quality requirements take into account how bias affects performance and fairness.

Detection:

  • A.6.2 AI system life cycle: Verification and validation also assess the system for impacts on individuals and groups, with test data that reflects the domain of use. Monitoring in operation continues this check.

Response:

  • A.9 Use of AI systems: Fairness is one of the objectives of responsible use. Human reviewers may override the system’s decisions, and the system is used only within its intended use.
  • A.6.2 AI system life cycle: Identified bias is removed through adjustments to data or model and a tested update.

Under Article 10 of the AI Act (Regulation (EU) 2024/1689), providers of high-risk systems examine their data for possible biases that could affect health, safety or fundamental rights or lead to prohibited discrimination, and they take measures to detect, prevent and mitigate them. Article 15(4) requires safeguards against feedback of biased outputs in systems that keep learning. Many areas of use in Annex III, such as employment, education and creditworthiness, count as high-risk precisely because of the risk of discrimination. Public bodies and private providers of public services that deploy high-risk systems from Annex III, as well as deployers of systems for credit scoring and for pricing in life and health insurance, carry out a fundamental rights impact assessment under Article 27 before use. In Germany, the General Equal Treatment Act (AGG) additionally prohibits discrimination, for example in working life (→ EU AI Act).

Sources

Frequently asked questions

Is every difference in treatment by an AI system discrimination?

No. A credit model is supposed to distinguish between people who are more and less able to pay. It becomes a problem when differences hinge on protected characteristics such as sex, origin, age or disability, or cannot be justified on objective grounds. This is exactly the line the impact assessment should draw.

Which fairness metric is the right one?

That depends on the use. Equal selection rates, equal error rates and equal precision per group generally cannot all be achieved at once. So decide which yardstick is appropriate for your purpose, and justify and document the choice.

What can deployers do if they do not train the model themselves?

Before use, they check whether the system was tested for their target group and ask the provider for results per group. In operation, they analyse results by group as far as the law allows and make sure there is a human review that can correct decisions. Certain deployers also have to carry out a fundamental rights impact assessment under Article 27 of the AI Act.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.