A logistics company uses image recognition to pull damaged parcels off the conveyor belt. In the test warehouse it works reliably. In the winter season, parcels arrive with traces of snow, plastic wrapping and handwritten labels, and the hall lighting flickers at night. The share of parcels rejected by mistake triples, although nobody changed anything in the system.
ISO/IEC 42001 lists the complexity of the environment in Annex C as a risk source for AI systems. The model alone is rarely the cause. The trouble starts because the world the system works in has more variations than the data it was developed and tested with.
What is behind it?
An AI system learns from examples. It is only reliable for the situations that appear often enough in those examples. The broader the field of application, the larger the space of possible situations, and the harder it becomes to predict how the system will react in a given case. The performance measured in testing then holds for only part of real operation.
Where complexity comes from
- Variety of inputs. Languages, dialects, image quality, document formats, sensors from different manufacturers. Every variant missing from training is a potential failure case.
- Changing conditions. Time of day, weather, season, workload. The environment changes while the system stays the same.
- Open interaction with people. A chatbot cannot know in advance which questions it will get. Users also react to the system’s answers and so change the inputs that come next.
- Chained systems. When the output of one AI system is the input of another, the uncertainties multiply. A small error at the start of the chain can trigger a wrong decision at the end.
Impact
What suffers first is the reliability of the results. In harmless applications that means rework. In systems that decide about people or control machines, health, safety and fundamental rights can be at stake. The treacherous part: the drop in performance often goes unnoticed, because the system answers with full confidence even in unfamiliar situations.
Practical examples
Speech recognition in a call centre. An insurer has calls transcribed and assigned to a request type automatically. For callers with a strong dialect or a noisy background, recognition drops sharply. Their cases end up in the wrong queue and handling is delayed, and the people affected are mostly older customers from rural areas.
Assistance system in manufacturing. A camera system inspects weld seams. After the move to a new hall with different lighting, it reports numerous defects on flawless parts. Because staff start ignoring the alerts after a few days, a batch that really is defective slips through later on.
Generative assistant with an open topic range. A company gives its staff a chat assistant for every question of daily work. It is well prepared for questions about internal processes and poorly prepared for legal and tax questions. Its answers sound equally confident in both cases.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.5 Assessing impacts of AI systems: The assessment asks in which situations the system will be used and what an error would mean there.
- A.6.1 Management guidance for AI system development: Development objectives and processes require the operating environment to be considered from the start.
- A.6.2 AI system life cycle: Requirements, verification and validation cover the described operating conditions before the system goes live.
Detection:
- A.6.2 AI system life cycle: Monitoring in operation reports when inputs or results move away from the tested conditions.
Response:
- A.9 Use of AI systems: A defined intended use and procedures for responsible use set out what happens when someone wants to use the system outside its limits.
Link to the AI Act
For high-risk systems, the AI Act (Regulation (EU) 2024/1689) takes up the topic in several places. Risk management under Article 9 looks at risks in intended use and in reasonably foreseeable misuse. Article 15 asks for an appropriate level of accuracy and robustness throughout the life cycle. The instructions for use under Article 13 state the conditions under which the declared performance holds, and deployers use the system in line with those instructions under Article 26 (→ EU AI Act).
Related threats
- T07 Immature technology / model drift: When the environment shifts over time, complexity turns into drift.
- T04 Poor data quality: Data that does not reflect the operating environment is the most common cause of gaps.
- T03 Excessive automation (automation bias): The less people step in, the longer an error in unfamiliar situations stays undetected.
Sources
- ISO/IEC 42001:2023, Annex C.3.1: Complexity of environment
- Regulation (EU) 2024/1689 (AI Act), Articles 9, 13, 15 and 26
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems