Zum Hauptinhalt springen
AI threat · ISO/IEC 42001 Annex C

T01 Complexity of environment

Updated on 4 min Reviewed by: Cenedril Editorial
ISO/IEC 42001EU AI Act

A logistics company uses image recognition to pull damaged parcels off the conveyor belt. In the test warehouse it works reliably. In the winter season, parcels arrive with traces of snow, plastic wrapping and handwritten labels, and the hall lighting flickers at night. The share of parcels rejected by mistake triples, although nobody changed anything in the system.

ISO/IEC 42001 lists the complexity of the environment in Annex C as a risk source for AI systems. The model alone is rarely the cause. The trouble starts because the world the system works in has more variations than the data it was developed and tested with.

What is behind it?

An AI system learns from examples. It is only reliable for the situations that appear often enough in those examples. The broader the field of application, the larger the space of possible situations, and the harder it becomes to predict how the system will react in a given case. The performance measured in testing then holds for only part of real operation.

Where complexity comes from

  • Variety of inputs. Languages, dialects, image quality, document formats, sensors from different manufacturers. Every variant missing from training is a potential failure case.
  • Changing conditions. Time of day, weather, season, workload. The environment changes while the system stays the same.
  • Open interaction with people. A chatbot cannot know in advance which questions it will get. Users also react to the system’s answers and so change the inputs that come next.
  • Chained systems. When the output of one AI system is the input of another, the uncertainties multiply. A small error at the start of the chain can trigger a wrong decision at the end.

Impact

What suffers first is the reliability of the results. In harmless applications that means rework. In systems that decide about people or control machines, health, safety and fundamental rights can be at stake. The treacherous part: the drop in performance often goes unnoticed, because the system answers with full confidence even in unfamiliar situations.

Practical examples

Speech recognition in a call centre. An insurer has calls transcribed and assigned to a request type automatically. For callers with a strong dialect or a noisy background, recognition drops sharply. Their cases end up in the wrong queue and handling is delayed, and the people affected are mostly older customers from rural areas.

Assistance system in manufacturing. A camera system inspects weld seams. After the move to a new hall with different lighting, it reports numerous defects on flawless parts. Because staff start ignoring the alerts after a few days, a batch that really is defective slips through later on.

Generative assistant with an open topic range. A company gives its staff a chat assistant for every question of daily work. It is well prepared for questions about internal processes and poorly prepared for legal and tax questions. Its answers sound equally confident in both cases.

Effective controls

The following ISO/IEC 42001 controls counter this threat.

Prevention:

Detection:

Response:

  • A.9 Use of AI systems: A defined intended use and procedures for responsible use set out what happens when someone wants to use the system outside its limits.

For high-risk systems, the AI Act (Regulation (EU) 2024/1689) takes up the topic in several places. Risk management under Article 9 looks at risks in intended use and in reasonably foreseeable misuse. Article 15 asks for an appropriate level of accuracy and robustness throughout the life cycle. The instructions for use under Article 13 state the conditions under which the declared performance holds, and deployers use the system in line with those instructions under Article 26 (→ EU AI Act).

Sources

Frequently asked questions

How do I tell that the environment of my AI system is complex?

By the range of situations the system can meet. The more different people, places, lighting conditions, languages, document types or workflows occur, and the less of this can be listed in advance, the more complex the environment. A simple test: can you compare the list of cases the system was tested for with the list of cases it meets in operation? If you cannot, you lack the overview.

Does T01 concern only providers or deployers as well?

Both. The provider decides which environments the system is built and tested for and writes that into the instructions for use. The deployer runs it in its real environment and has to check whether that environment matches the instructions. Most of the damage happens exactly at this seam.

Does more training help against a complex environment?

Partly. More and more varied data improves coverage, yet the environment stays as complex as it was. What works better is a combination of clearly described operating limits, tests across those limits, human oversight for unclear cases and monitoring in operation that reports new situations early.

No Cookies!

This wiki collects nothing, bakes nothing and leaves nothing behind. There's nothing to consent to. Privacy doesn't get better than this.