Zum Hauptinhalt springen
Misuse pattern · AI Act

MU-06 Use on untested groups

Updated on 6 min Reviewed by: Cenedril Editorial
EU AI ActISO/IEC 42001

A dermatology practice in a large city uses software that rates images of moles and marks conspicuous findings. It was developed and tested mostly on images of light skin. The practice uses it for all patients, because the contract says nothing on the matter. On dark skin, the system misses findings it would reliably mark on light skin. Nobody knows the error rate for these people, and they rely on the system just as much as everyone else.

This pattern describes applying an AI system to people for whom it was neither developed nor tested. The AI Act (Regulation (EU) 2024/1689) treats such cases as reasonably foreseeable misuse under Article 3(13), because every organization deals with people who differ from the average of the training data.

What is behind it?

An AI system learns from the people who appear in its data. Every performance figure holds for the groups that were sufficiently represented in training and testing, and for nobody else with certainty. If the system is applied to other people, such as children, older persons, people with disabilities, other language groups or skin types, there is no statement on its reliability. Performance there is often worse, and systematically so.

How it happens in daily work

  • The circle grows. A system is used in new countries, for new customer groups or in another language without being tested again.
  • The information is missing. The documents do not say which groups were tested, and the deployer does not ask.
  • The average reassures. Good overall accuracy hides gaps in small groups.
  • The demo misleads. The decision for the system was made after a demonstration with the organization’s own staff, who resemble its customers very little.
  • Special groups come along. Children, older people or people with disabilities end up within scope because nobody drew a line.

Why it is risky

Errors that hit one group more often are biases in the sense of threat T12 and can lead to unlawful discrimination. They often hit precisely those who are least able to defend themselves. And they stay invisible for a long time, because membership of a group is usually not recorded in the deployer’s data. Looking only at the overall figures reveals nothing.

Impact

The impact ranges from poorer service to refused benefits, missed diagnoses and disadvantageous decisions in education, employment or lending. A random error hits individuals, a systematic error hits an entire group again and again. Besides the harm to the people affected, this raises the accusation of discrimination.

Practical examples

Citizens’ chat in official language. A municipality introduces a chat assistant for applications, tested with requests in standard German. A large share of users write in simple German or with many errors. The assistant understands these requests less well and refers them to the wrong forms more often.

Reading software in primary school. A school uses software that assesses reading performance and proposes support groups. It was tested on children without a reading and spelling disorder. Children with dyslexia are systematically rated too low and end up in groups that do not challenge them.

Aptitude test for apprentices. A company uses a test that was developed and validated for university graduates for apprenticeship applications as well. Some applicants are minors, and the tasks assume experience they cannot have yet. The results mainly reflect the applicants’ age.

Effective controls

The following ISO/IEC 42001 controls counter this pattern.

Prevention:

  • A.5 Assessing impacts of AI systems: The assessment considers the consequences for individuals and groups and takes into account the specific protection needs of, for example, children, older people, people with disabilities and workers. It also documents which demographic groups the system applies to.
  • A.8 Information for interested parties: Users learn which groups the system has been tested for and where its performance drops.
  • A.9 Use of AI systems: Use follows the system’s documents, and the data it is applied to matches what those documents describe.

Detection:

  • A.6.2 AI system life cycle: Verification and validation cover the relevant groups, and monitoring in operation checks whether the system works with real data as expected.

Response:

For deployers who have to carry out a fundamental rights impact assessment, the AI Compliance Starter Kit offers a template under Article 27. The AI Usage Policy requires in Section 4.4 that outputs about people are checked for bias, and in Section 7 that results which affect people unfairly are reported. Section 10 takes up the duty of representative input data for high-risk systems.

For high-risk AI systems, the Regulation addresses this in several places. The provider evaluates foreseeable misuse under Article 9(2)(b) and considers possible adverse effects on minors and other vulnerable groups under Article 9(9). Its training, validation and testing data must be representative of the persons and groups the system is intended for (Article 10(3) and (4)). In the instructions for use, it states, where applicable, the performance regarding specific persons or groups (Article 13(3)(b)(v)) and the circumstances in which misuse may lead to risks (point (iii)).

Under Article 26(4), the deployer makes sure that input data under its control is relevant to the intended purpose and sufficiently representative. Certain deployers, such as public bodies, private entities providing public services and deployers that assess creditworthiness or price life and health insurance, describe the affected groups of people and their specific risks of harm in the fundamental rights impact assessment under Article 27(1)(c) and (d) (→ EU AI Act).

Sources

Frequently asked questions

How do I find out which groups an AI system was tested on?

For high-risk systems, the instructions for use state, where applicable, the performance regarding specific persons or groups (Article 13(3)(b)(v) of the AI Act) and the input data specifications. For other systems, the only way is to ask the provider before buying. If no answer comes, treat the group as untested.

Is poorer performance for one group automatically discrimination?

Not always in the legal sense, though it is a serious warning sign. If the system systematically disadvantages a group in an area such as employment, education, credit or health, unlawful discrimination is close at hand. That is why the question belongs in the impact assessment before the system is applied to the group.

May I analyse results by group to find gaps?

Often yes, with care. Characteristics such as age or language can usually be analysed without difficulty. For special categories of personal data, such as health or ethnic origin, tight limits apply under the GDPR and the AI Act. Clarify the framework beforehand with whoever is responsible for data protection.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.