A dermatology practice in a large city uses software that rates images of moles and marks conspicuous findings. It was developed and tested mostly on images of light skin. The practice uses it for all patients, because the contract says nothing on the matter. On dark skin, the system misses findings it would reliably mark on light skin. Nobody knows the error rate for these people, and they rely on the system just as much as everyone else.
This pattern describes applying an AI system to people for whom it was neither developed nor tested. The AI Act (Regulation (EU) 2024/1689) treats such cases as reasonably foreseeable misuse under Article 3(13), because every organization deals with people who differ from the average of the training data.
What is behind it?
An AI system learns from the people who appear in its data. Every performance figure holds for the groups that were sufficiently represented in training and testing, and for nobody else with certainty. If the system is applied to other people, such as children, older persons, people with disabilities, other language groups or skin types, there is no statement on its reliability. Performance there is often worse, and systematically so.
How it happens in daily work
- The circle grows. A system is used in new countries, for new customer groups or in another language without being tested again.
- The information is missing. The documents do not say which groups were tested, and the deployer does not ask.
- The average reassures. Good overall accuracy hides gaps in small groups.
- The demo misleads. The decision for the system was made after a demonstration with the organization’s own staff, who resemble its customers very little.
- Special groups come along. Children, older people or people with disabilities end up within scope because nobody drew a line.
Why it is risky
Errors that hit one group more often are biases in the sense of threat T12 and can lead to unlawful discrimination. They often hit precisely those who are least able to defend themselves. And they stay invisible for a long time, because membership of a group is usually not recorded in the deployer’s data. Looking only at the overall figures reveals nothing.
Impact
The impact ranges from poorer service to refused benefits, missed diagnoses and disadvantageous decisions in education, employment or lending. A random error hits individuals, a systematic error hits an entire group again and again. Besides the harm to the people affected, this raises the accusation of discrimination.
Practical examples
Citizens’ chat in official language. A municipality introduces a chat assistant for applications, tested with requests in standard German. A large share of users write in simple German or with many errors. The assistant understands these requests less well and refers them to the wrong forms more often.
Reading software in primary school. A school uses software that assesses reading performance and proposes support groups. It was tested on children without a reading and spelling disorder. Children with dyslexia are systematically rated too low and end up in groups that do not challenge them.
Aptitude test for apprentices. A company uses a test that was developed and validated for university graduates for apprenticeship applications as well. Some applicants are minors, and the tasks assume experience they cannot have yet. The results mainly reflect the applicants’ age.
Effective controls
The following ISO/IEC 42001 controls counter this pattern.
Prevention:
- A.5 Assessing impacts of AI systems: The assessment considers the consequences for individuals and groups and takes into account the specific protection needs of, for example, children, older people, people with disabilities and workers. It also documents which demographic groups the system applies to.
- A.8 Information for interested parties: Users learn which groups the system has been tested for and where its performance drops.
- A.9 Use of AI systems: Use follows the system’s documents, and the data it is applied to matches what those documents describe.
Detection:
- A.6.2 AI system life cycle: Verification and validation cover the relevant groups, and monitoring in operation checks whether the system works with real data as expected.
Response:
- A.5 Assessing impacts of AI systems: If a gap appears, the assessment is updated, use for the group concerned is limited and the provider is informed.
For deployers who have to carry out a fundamental rights impact assessment, the AI Compliance Starter Kit offers a template under Article 27. The AI Usage Policy requires in Section 4.4 that outputs about people are checked for bias, and in Section 7 that results which affect people unfairly are reported. Section 10 takes up the duty of representative input data for high-risk systems.
Link to the AI Act
For high-risk AI systems, the Regulation addresses this in several places. The provider evaluates foreseeable misuse under Article 9(2)(b) and considers possible adverse effects on minors and other vulnerable groups under Article 9(9). Its training, validation and testing data must be representative of the persons and groups the system is intended for (Article 10(3) and (4)). In the instructions for use, it states, where applicable, the performance regarding specific persons or groups (Article 13(3)(b)(v)) and the circumstances in which misuse may lead to risks (point (iii)).
Under Article 26(4), the deployer makes sure that input data under its control is relevant to the intended purpose and sufficiently representative. Certain deployers, such as public bodies, private entities providing public services and deployers that assess creditworthiness or price life and health insurance, describe the affected groups of people and their specific risks of harm in the fundamental rights impact assessment under Article 27(1)(c) and (d) (→ EU AI Act).
Related patterns and threats
- T12 Bias and discrimination: The threat this pattern triggers most often.
- T04 Poor data quality: Data in which a group is missing is the underlying cause.
- MU-01 Use outside the intended purpose: The overarching pattern, here with a shifted group of people.
- MU-12 Use outside the operating limits: Related when the limit concerns a language or data quality.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3(13), Articles 9, 10, 13, 26 and 27
- ISO/IEC 42001:2023, Annex B.5.3, B.5.4, B.6.2.4 and B.9.4
- AI Usage Policy of the AI Compliance Starter Kit, Sections 4.4, 7 and 10
- Fundamental Rights Impact Assessment: template of the AI Compliance Starter Kit