A wholesaler has an AI system check incoming invoices and propose them for payment. Accounts payable approves the proposals, at first by spot check, after a few months with a single click for the whole batch. A forged invoice with changed bank details goes through, because the system marked it as plausible and nobody looks closely any more. On paper there was a human approval; in reality the team had long since handed its judgement over to the system.
ISO/IEC 42001 lists the level of automation as a risk source in Annex C, because it touches safety, fairness and security. In the product, the threat carries the name of its most common mechanism: automation bias, the tendency to trust automatically generated results more than one’s own review.
What is behind it?
Automation shifts the human role from deciding to monitoring. That sounds like relief, yet it demands a particular kind of attention that is hard to sustain in daily work. The more reliably a system works most of the time, the less people expect it to fail. That is exactly when the rare error slips through.
Levels of automation
- The AI supports, a person decides. The system provides hints or suggestions. Danger: the suggestion anchors the judgement, and the person looks for confirmation rather than contradiction.
- The AI decides, a person monitors. The system acts on its own, and a person can step in. Danger: monitoring becomes a formality because intervention rarely seems necessary.
- The AI decides alone. Nobody is involved in the individual case. Danger: errors take effect immediately and at scale, and often surface only through complaints.
Why people trust too much
- Time pressure and high volumes. Someone who reviews hundreds of suggestions a day cannot recheck each one.
- Missing information. The interface shows a result with neither uncertainty nor reasoning. What is a doubt supposed to latch onto?
- Diffused responsibility. Everyone assumes the system or another unit has already checked.
- Skewed incentives. Whoever departs from the suggestion has to justify it. Whoever agrees has to do nothing.
Impact
What is at stake depends on the decision. In scoring and recruitment, people can be disadvantaged systematically without anyone noticing. In IT security, an automated system may lock legitimate accounts or let an attack pass because the alert drowns in the volume. In technical plants, physical safety can be at stake. All these cases share one feature: automation multiplies an error. Then there is the legal risk. Under Article 22 GDPR, decisions with legal or similarly significant effects may be taken solely by automated means only in exceptional cases.
Practical examples
Preliminary credit decision. A scoring system recommends acceptance or rejection. Case handlers follow the recommendation in almost every case, since a departure requires written reasoning and the team lead’s consent, while agreement requires nothing. In practice the system decides alone, although the process provides for a human decision.
Automatic account lockouts. An IT operations tool locks user accounts by itself when login behaviour looks unusual. At quarter end the finance team works late into the night, and the system locks dozens of accounts. Because nobody has questioned the lockouts in months, it takes hours before someone spots the cause and unblocks the closing.
Ranking in recruitment. A recruitment system sorts applications by fit. Under time pressure, recruiters only look at the top ten. Anyone ranked further down is never read by a human, even when the rating stems from an error in the CV import.
Effective controls
The following ISO/IEC 42001 controls counter this threat.
Prevention:
- A.9 Use of AI systems: The organization sets objectives for responsible use, including human oversight. It determines where people review results and may override the system’s decisions, and it asks whether automated decision-making is appropriate for the purpose at all.
- A.6.2 AI system life cycle: Requirements and design already settle how people interact with the system and how results are presented.
- A.8 Information for interested parties: Users learn how accurate the system is, when human oversight is needed and how and when they can override it.
Detection:
- A.6.2 AI system life cycle: Monitoring in operation tracks performance and errors, and event logs show which results were accepted.
- A.9 Use of AI systems: Staff report concerns about individual outputs and about changes in performance.
Response:
- A.9 Use of AI systems: The processes for responsible use set out what happens when the review has become a formality: less automation, smaller case volumes, more training or a different cut of the task.
Link to the AI Act
The AI Act (Regulation (EU) 2024/1689) addresses the issue directly. Article 14 requires that high-risk systems can be overseen effectively by people, and Article 14(4)(b) calls automation bias by its name: the people in charge of oversight should stay aware of the tendency to rely too heavily on the outputs. The system has to enable them to interpret the outputs correctly, to disregard or override them and to halt operation. Under Article 26(2), deployers assign oversight to people with the necessary competence, training and authority, and Article 4 requires sufficient AI literacy among staff. For decisions about people, Article 22 GDPR applies as well. In 2023 the Court of Justice of the EU ruled (Case C-634/21) that even the automated calculation of a score can fall under it when a third party relies heavily on it for its decision (→ EU AI Act).
Related threats
- T13 Hallucination / faulty output: Fabricated content becomes dangerous above all when it passes on unchecked.
- T02 Lack of transparency and explainability: Without reasoning, a doubt has nothing to hold on to.
- MU-02 Adopting outputs unchecked: The misuse pattern describes the same weakness from the angle of usage rules.
Sources
- ISO/IEC 42001:2023, Annex C.3.3: Level of automation
- Regulation (EU) 2024/1689 (AI Act), Articles 4, 14 and 26
- Regulation (EU) 2016/679 (GDPR), Article 22
- ISO/IEC 42001 in the Cenedril wiki: Annex C and the risk sources for AI systems